跳到正文
arXiv cs.AI· Simon Yu, Nicholas Tomlin, Marwa Abdulhai, Ximing Lu, Derek Chong, Abe Hou, Dilara Soylu, Sergey Levine, Christopher D. Manning, Weiyan Shi·· 4 小时前AI 评分49

单一冻结模拟器不够:多智能体 RL 中的模拟器坍缩问题

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

AI 导读

研究发现,用单个 LLM 模拟用户来训练多智能体强化学习智能体会导致"模拟器坍缩",习得的策略难以泛化到未见模拟器和真实用户。

正文 · 原文

View PDF HTML (experimental)

Abstract:Multi-agent reinforcement learning (MARL) for human-AI interaction usually trains the agent against a single large language model (LLM) that simulates the user. We show that agents trained this way can systematically fail to generalize, and we trace the failure to simulator collapse: the simulator LLM is mode-collapsed, and a policy trained against it learns narrow strategies that exploit the simulator's mode behavior; these strategies transfer poorly to unseen simulators and to real users. We formalize the collapse and propose two solutions. Verbalized Sampling, applied at inference time, samples each simulator reply from a verbalized response distribution. Co-Training, applied during training similar to self-play, co-evolves the simulator as the policy, so the policy cannot exploit the mode behavior. A population variant samples the simulator for each rollout from a pool of recent checkpoints. We validate both solutions on three benchmarks: Persuasion for Good, tau^2-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions also preserve the policy diversity that collapses under single-simulator RL. Overall, our findings on simulator collapse point to diverse simulators as a requirement for RL agents that serve real users. We release SCOPE, an open-source framework for Population Co-Training in multi-agent RL.
Comments: 44 pages, 29 figures
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2608.12253 [cs.CL]
  (or arXiv:2608.12253v3 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2608.12253

arXiv-issued DOI via DataCite

Submission history

From: Simon Yu [view email]
[v1] Wed, 12 Aug 2026 16:55:50 UTC (2,528 KB)
[v2] Mon, 17 Aug 2026 17:48:56 UTC (2,548 KB)
[v3] Tue, 17 Nov 2026 07:18:04 UTC (2,507 KB)

来源:arXiv cs.AI · arxiv.org