跳到正文
arXiv cs.AI· Seungyub Han, Taehyun Cho, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee·· 10 小时前AI 评分31

MINT:面向多步离策略 RL 的截断会合时间指标

MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

AI 导读

研究者提出 MINT(Meeting-time INdicators for Truncation),用二元停止指标替代多步离策略时序差分中的逐步迹系数乘积,在真实值函数下其幂等性带来由会合时间存活率约束的方差界。

正文 · 原文

View PDF HTML (experimental)

Abstract:Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.
Comments: Accepted at NeurIPS 2026. 81 pages, 40 figures, including appendices. Code: this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24413 [cs.LG]
  (or arXiv:2611.24413v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.24413

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Seungyub Han [view email]
[v1] Wed, 18 Nov 2026 13:40:04 UTC (2,788 KB)

来源:arXiv cs.AI · arxiv.org