arXiv cs.AI· Xingyu Su, Abhishek Kumar, Qing Ping, Youzhi Luo, Jonathan Buck, Zach Zhang, Subramanian Chidambaram, Vinayak Arannil·· 12 小时前AI 评分38
从自蒸馏到自练习：多轮智能体的特权信息方法
From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents
AI 导读
研究指出，on-policy 自蒸馏（OPSD）在多轮智能体中会让模型表现得像拥有从未见过的特权信息，性能甚至可能低于未训练基座模型。为此提出 Privileged Self-Practice（PSP），将特权信息从损失函数移到采样器：当任务 rollout 大多失败时，注入分析模型撰写的任务指令重新采样，并用不变的 GRPO 目标训练。
正文 · 原文
Abstract:On-policy self-distillation (OPSD) has become a popular recipe for post-training LLM agents. It supervises the agent model at the token level with a stronger teacher view of the same model, obtained by conditioning on privileged information (PI). In this work, we show that in multi-turn agents, this paradigm teaches the student to act with confidence but without the information behind it. The trained agent behaves as if it had privileged information it never observed, and its performance falls well short of plain RL, in the worst case below the untrained base model. Therefore, we propose Privileged Self-Practice (PSP), which keeps the PI and moves it from the loss to the sampler. When the student's rollouts on a task mostly fail, we inject a short per-task instruction written by an analyzer model, sample the task again with the instruction in context, and train on the result with an unchanged GRPO objective. The privileged information stays in the prompt and never enters the loss. Across AppWorld and SWE-bench Verified, with three different student models, PSP obtains the best average score in every setting and is the only method that consistently outperforms plain GRPO, with relative improvements over GRPO of 7% to 58% in average score across settings. Project page: this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.29051 [cs.AI] |
| (or arXiv:2609.29051v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.29051 arXiv-issued DOI via DataCite |
Submission history
From: Xingyu Su [view email]
[v1]
Thu, 24 Sep 2026 05:32:25 UTC (409 KB)
[v2]
Thu, 15 Oct 2026 19:31:21 UTC (411 KB)
来源：arXiv cs.AI · arxiv.org