arXiv cs.CL· Hojun Cho, Kang Eun Jeon, Jaegul Choo·· 11 小时前AI 评分31
Adaptive-Policy Distillation：信息不对称下的知识注入方法
Adaptive-Policy Distillation: Knowledge Injection under Information Asymmetry
AI 导读
研究者提出 Adaptive-Policy Distillation（APD），在无标签、无验证器条件下将知识注入效果较 on-policy distillation（OPD）最高提升 54.6%，在 QASPER 上取得 0.558 F1、BioASQ 上取得 0.486 MRR 的最佳成绩。
正文 · 原文
Abstract:On-policy distillation (OPD), which uses reverse-KL supervision on student rollouts to transfer a context-conditioned model's behavior into student model parameters for closed-book inference, has shown strong results. However, we find that this success does not translate to knowledge injection: across two document-QA benchmarks, OPD injects document knowledge far less efficiently than simple off-policy distillation. We trace this reversal to two failures that are structural, yet sparse across token positions. Reverse KL weights its gradient by the student distribution and therefore barely lifts document-specific tokens the student considers unlikely; after forward KL restores this signal, student rollouts expose later positions on the target trajectory only after generating the preceding tokens correctly, so coverage deteriorates with target length. Motivated by the concentration of teacher-student divergence at these hard positions, we propose Adaptive-Policy Distillation (APD), whose per-token gate jointly shifts the rollout toward the teacher and the objective toward forward KL where needed, withdrawing both interventions as the student learns. Without labels or a verifier, APD improves over OPD by up to 54.6%, achieving the best scores of 0.558 F1 on QASPER and 0.486 MRR on BioASQ.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.00852 [cs.CL] |
| (or arXiv:2611.00852v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00852 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hojun Cho [view email]
[v1]
Sat, 31 Oct 2026 11:59:35 UTC (100 KB)
来源：arXiv cs.CL · arxiv.org