arXiv cs.CL· Bangji Yang, Jiajun Fan, Zhaoyu Zhu, Hongbo Ma, Ge Liu·· 2 天前AI 评分32
无需训练的策略蒸馏：仅用前向传播融合专家模型
Training-Free On-Policy Distillation
AI 导读
研究者提出无需训练的策略蒸馏（training-free on-policy distillation），仅靠前向传播即可将多个独立微调的专家模型能力聚合为一个模型，平均精度超过任一单一专家。
正文 · 原文
Abstract:On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation (\method). What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo~3 7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks.
| Comments: | 20 pages, 6 figures, 9 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22236 [cs.LG] |
| (or arXiv:2610.22236v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22236 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongbo Ma [view email]
[v1]
Thu, 15 Oct 2026 17:15:01 UTC (314 KB)
来源：arXiv cs.CL · arxiv.org