arXiv cs.AI· Deyang Kong, Zhaoyi Li, Sijian Ren, Ying Wei, Defu Lian, Shikun Zhang, Wei Ye·· 10 小时前AI 评分32
Agentic On-Policy Distillation 如何隐式重塑 LLM 的 ReAct 交互模式
Agentic On-Policy Distillation Implicitly Reshapes LLMs' ReAct Patterns
AI 导读
研究揭示 on-policy distillation(OPD)会跨域重塑 LLM 智能体的 ReAct 交互模式:通用推理监督将数学工具调用尝试从 70.4% 降至 14.4%,准确率从 16.7% 升至 38.0%;而 ALFWorld 监督使 τ²-bench 准确率从 41.5% 跌至 6.6%。
正文 · 原文
Abstract:On-policy distillation (OPD) transfers specialized capabilities to language-model agents, but its cross-domain effects on interaction remain unclear. We investigate how OPD reshapes ReAct interaction patterns across mathematics, ALFWorld, and $\tau^2$-bench, focusing on reasoning, actions, and observation acquisition. General-reasoning supervision reduces mathematical tool attempts from 70.4% to 14.4% while increasing accuracy from 16.7% to 38.0%; code-interpreter supervision improves household actions and retail queries. In contrast, ALFWorld supervision reduces $\tau^2$-bench accuracy from 41.5% to 6.6%, accompanied by frequent tool calls that violate user-information dependencies. To address negative transfer using only source-domain training trajectories, we explore selective OPD guided by teacher preference changes before and after domain-specific RL. Retaining the top 95% of positions raises $\tau^2$ accuracy from 11.5% to 29.9% with a source-learning trade-off.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.19244 [cs.AI] |
| (or arXiv:2611.19244v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.19244 arXiv-issued DOI via DataCite |
Submission history
From: Deyang Kong [view email]
[v1]
Sat, 14 Nov 2026 08:18:45 UTC (1,359 KB)
[v2]
Wed, 18 Nov 2026 05:32:45 UTC (1,359 KB)
来源:arXiv cs.AI · arxiv.org