arXiv cs.AI· Kintan Saha, Hamed Masoudi, Xinyi Sheng, Dominik Baumann, Joni Pajarinen, Arsenii Mustafin·· 2 天前AI 评分22
V-SHARE：基于优势的强化学习算法的可证明收敛保证
V-SHARE: Provable Convergence Guarantees for Advantage-based Algorithms
AI 导读
研究者提出基于优势的价值学习算法 V-SHARE，将动作价值函数分解为状态价值与动作优势并跨动作共享信息，结合重要性加权的 Bellman 残差修正，首次给出该类算法的完整理论刻画，涵盖渐近收敛与非渐近有限样本保证。
正文 · 原文
Abstract:In reinforcement learning, advantage-based value-learning methods, unlike standard Q-learning, share information across actions by decomposing the action-value function into a state-dependent value component and action-specific advantages. While this mechanism has proven effective in both tabular and deep reinforcement learning, its theoretical understanding remains incomplete. We introduce V-SHARE, an advantage-based value-learning algorithm that combines cross-action value sharing with an importance-weighted Bellman-residual correction. To our knowledge, we provide the first complete theoretical characterization of an advantage-based algorithm, including both asymptotic convergence and non-asymptotic finite-sample guarantees. We identify a regime in which V-SHARE mitigates the rare-action bottleneck of asynchronous Q-learning and uncover a contraction--variance tradeoff between stronger deterministic contraction and increased importance-weighting noise. Motivated by this tradeoff, we develop decreasing residual-preconditioning schedules that combine faster early contraction with lower asymptotic noise. We further develop a two-timescale analysis that removes the single-timescale residual-weight threshold while preserving convergence guarantees. Experiments on discrete Markov decision processes (MDPs) and deep reinforcement-learning benchmarks corroborate our results.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.21448 [cs.LG] |
| (or arXiv:2610.21448v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21448 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kintan Saha [view email]
[v1]
Thu, 15 Oct 2026 07:52:35 UTC (359 KB)
来源：arXiv cs.AI · arxiv.org