arXiv cs.AI· Seungyub Han, Taehyun Cho, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee·· 10 小时前AI 评分31
MINT:面向多步离策略 RL 的截断会合时间指标
MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL
AI 导读
研究者提出 MINT(Meeting-time INdicators for Truncation),用二元停止指标替代多步离策略时序差分中的逐步迹系数乘积,在真实值函数下其幂等性带来由会合时间存活率约束的方差界。
正文 · 原文
Abstract:Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.
| Comments: | Accepted at NeurIPS 2026. 81 pages, 40 figures, including appendices. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24413 [cs.LG] |
| (or arXiv:2611.24413v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24413 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Seungyub Han [view email]
[v1]
Wed, 18 Nov 2026 13:40:04 UTC (2,788 KB)
来源:arXiv cs.AI · arxiv.org