arXiv cs.CL· Kejian Zhu, Zhuoran Jin, Shangqing Tu, Tianyi Men, Yupu Hao, Jiachun Li, Kang Liu, Jun Zhao·· 12 小时前AI 评分38
多轮强化学习究竟提升了 LLM 智能体的什么能力？
What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?
AI 导读
研究通过 Pass@(K,T) 在采样广度 K 与交互深度 T 上评估发现，多轮智能体强化学习的轨迹级收益依赖充足的轮次预算，预算收紧时收益显著衰减。该方法的优势集中在错误恢复与长程韧性，而非均匀提升单步决策质量。作者提出 BudgetMix RL，在训练中变化交互预算，在多种执行设置下取得稳健表现。
正文 · 原文
Abstract:While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth $K$ and interaction depth $T$ via $Pass@(K,T)$, we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: this https URL
| Comments: | Project Page: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.06483 [cs.LG] |
| (or arXiv:2611.06483v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06483 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kejian Zhu [view email]
[v1]
Wed, 4 Nov 2026 13:24:54 UTC (956 KB)
来源：arXiv cs.CL · arxiv.org