arXiv cs.CL· Yilong Xu, Minrui Xu, Xinyu Zhou, Xiang Long, Haochen Tan, Zhijiang Guo·· 2 天前AI 评分27
Mosaic：多会话智能体强化学习
Mosaic: Multi-Session Agentic Reinforcement Learning
AI 导读
针对现有智能体强化学习仅假设单会话轨迹、与上下文压缩和子智能体分叉等推理时执行方式不匹配的问题，研究者提出多会话智能体强化学习设置及 Mosaic 方法，采用无 critic 的分层优化，在轨迹层引入会话使用信号、在会话层按估计边际贡献调整优势。在 BrowseComp 上以 Qwen3.5-4B 测试，Mosaic 在上下文压缩和子智能体分叉场景均优于广播式方法，且平均使用会话数更少。
正文 · 原文
Abstract:Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested is a promising target for future advances in multi-session agentic RL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.22174 [cs.CL] |
| (or arXiv:2610.22174v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22174 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yilong Xu [view email]
[v1]
Thu, 15 Oct 2026 16:32:44 UTC (818 KB)
来源：arXiv cs.CL · arxiv.org