arXiv cs.AI· Zixu Wang, Yu Zhao, Xing Zheng, Bingbing Xu, Tian Xueyun, JiaTang Luo, Weiqiang Wang, Bowen Song, Huawei Shen, Xueqi Cheng·· 1 天前AI 评分31
OPSD4MAS：用On Policy Self Distillation将多智能体通信转为Token级信用
Turning Multi-Agent Communication into Token-Level Credit with On Policy Self Distillation
AI 导读
OPSD4MAS 通过 On Policy Self Distillation（OPSD）把多智能体系统中的未来通信转化为细粒度 token 级信用，先由外部模型抽取事实并评估轨迹关系做粗筛，再用 leave-one-out 过滤目标 token 影响。
正文 · 原文
Abstract:Reinforcement learning improves problem solving in large language model multi-agent systems. Existing approaches overlook communication knowledge or compress it into scalar rewards, leaving coarse credit with little guidance on which decisions each agent should reinforce or revise. OPSD4MAS converts future communication into fine-grained token-level credit through On Policy Self Distillation (OPSD). An external model extracts facts and assesses their trajectory relations for coarse-grained filtering, followed by fine-grained leave-one-out filtering of target-token influence against agent-wise outcome advantages. The fact-conditioned policy supplies token-level credit, combined with GRPO through a linearly annealed joint objective. Our method improves mathematical pass@4 over GRPO by up to 20.00 percentage points. Ablations support complete filtering, while frozen-history injection diagnostics support fact utility. Together, these results support combining agent-level outcome guidance with communication-derived token-level credit. Code is available at this https URL.
| Comments: | 14pages,4 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.6; I.2.11; I.2.7 |
| Cite as: | arXiv:2611.16206 [cs.AI] |
| (or arXiv:2611.16206v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16206 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zixu Wang [view email]
[v1]
Thu, 12 Nov 2026 02:57:08 UTC (693 KB)
来源：arXiv cs.AI · arxiv.org