arXiv cs.CL· Yuhan Sun, Jiaxin Zhao, Yongdong Luo, Yibo Hu, Meiguang Jin, Junfeng Ma, Weixun Wang, Weihang Pan, Zhongan Bi·· 12 小时前AI 评分32
MultiAgent-OPD：用同策略蒸馏端到端训练分层多智能体系统
MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation
AI 导读
MultiAgent-OPD 将同策略蒸馏与强化学习结合，端到端联合训练共享参数的 Leader-Searcher-Answerer 团队。训练时冻结的 Teacher 为学生生成的搜索与摘要 token 打分，为 Searcher 提供局部过程监督，Leader 从团队成败中学习、Answerer 从候选答案正确性中学习。
正文 · 原文
Abstract:In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Jointly training the Leader and the subagents is difficult because a final answer gives only indirect feedback to the subagents. An LLM judge can score the performance of the subagents, but these scores depend on the judge's own assessment of the quality of subagents' responses. We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly train a shared-parameter Leader-Searcher-Answerer team end to end. We first train a Teacher in a Searcher-Answerer workflow using answer correctness. During team training, the frozen Teacher scores student-generated search and summary tokens to provide local process supervision for the Searchers, while the Leader learns from team success and the Answerer from candidate correctness. Across seven datasets, the trained system reaches 38.68% macro avg@4, exceeding Prompted Critic by 4.06 percentage points and C-MAS + RL, the trained cascaded multiagent system, by 2.61 percentage points. The team also solves more questions within four attempts.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.23234 [cs.CL] |
| (or arXiv:2610.23234v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23234 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhan Sun [view email]
[v1]
Fri, 16 Oct 2026 10:01:52 UTC (799 KB)
来源：arXiv cs.CL · arxiv.org