arXiv cs.AI· Prabhat Nagarajan, Martha White, Marlos C. Machado·· 10 小时前AI 评分31
Deep Double Q-learning：显式训练双 Q 函数的深度强化学习算法
Deep Double Q-learning
AI 导读
论文提出 Deep Double Q-learning（DDQL），通过 Double Q-learning 目标显式训练两个 Q 函数，并结合较低回放比、更长目标网络更新间隔与共享层稳定训练。
正文 · 原文
Abstract:Double Q-learning is a classical control algorithm that mitigates the maximization bias of Q-learning. To do so, it explicitly trains two independent action-value functions and uses them to decouple action-selection from action-evaluation when computing bootstrap targets. Double DQN adapts this decoupling to deep reinforcement learning (RL) but directly trains only a single action-value function and does not fully decouple its estimators. Consequently, the two estimators remain correlated, and overestimation persists. In this paper, we introduce Deep Double Q-learning (DDQL), a deep RL algorithm that explicitly trains two Q-functions through the Double Q-learning objective. DDQL stabilizes training through a combination of techniques, including lower replay ratios, longer target network update intervals, and shared layers. Across 57 Atari 2600 games, DDQL improves aggregate performance over Double DQN, outperforming it on 47 games while further reducing overestimation. We study key design choices of DDQL, including its network architecture, replay ratio, and minibatch sampling strategies.
| Comments: | Neural Information Processing Systems (NeurIPS), 2026; |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2507.00275 [cs.LG] |
| (or arXiv:2507.00275v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2507.00275 arXiv-issued DOI via DataCite |
Submission history
From: Prabhat Nagarajan [view email]
[v1]
Mon, 30 Jun 2025 21:32:46 UTC (5,968 KB)
[v2]
Thu, 14 May 2026 20:49:30 UTC (6,356 KB)
[v3]
Mon, 9 Nov 2026 21:12:50 UTC (7,574 KB)
来源：arXiv cs.AI · arxiv.org