arXiv cs.CL· Yiqiao Qiu·· 11 小时前AI 评分31
面向 LLM 持续学习、以多教师在线策略蒸馏实现回放
Replay for LLM Continual Learning with Multi-Teacher On-Policy Distillation
AI 导读
论文提出持续多教师在线策略蒸馏（MOPD）的回放方案：每个新阶段用上一阶段的学生模型作为教师回放早期领域提示词，无需保留早期专家或阶段快照。在逻辑、数学、代码、安全与指令遵循五领域、1.5B 学生模型的 14 项基准上，该方案无需存储任何快照即可弥合顺序蒸馏与联合 MOPD 的大部分差距；每个早期领域固定回放份额时，效果与用全部已存快照做恢复训练相当。
正文 · 原文
Abstract:Multi-teacher on-policy distillation (MOPD) distills several domain experts into one student and is used in the post-training of recent large language models. It trains on all domains at once, so every expert must be available at the same time. In practice, capabilities often arrive one after another, and the experts of earlier domains may be gone when a new domain is learned. Distilling the experts in sequence then suffers from catastrophic forgetting of the earlier domains. Some recent pipelines store a snapshot after every stage and end with a multi-teacher recovery stage taught by all snapshots. The number of stored models then grows with the number of stages. Different from these pipelines, this paper formulates continual MOPD by what is kept from earlier stages and studies replay toward the previous student. Inside every stage, prompts of earlier domains are replayed with the student of the previous stage as their teacher, so neither earlier experts nor stage snapshots are kept. Two design choices of replay are explored. The replay budget of a stage is either shared among the earlier domains or fixed per earlier domain. Replay steps either distill the token distribution alone or add alignment of hidden states, region-pooled self-attention scores and value relations. A five-domain benchmark of logic, math, code, safety and instruction following is built from public experts of one 1.5B student. Experiments on its 14 benchmarks show that, without storing any snapshot, replay toward the previous student closes most of the gap between sequential distillation and joint MOPD. With a fixed replay share per earlier domain, it performs on par with recovery from all stored snapshots.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.02746 [cs.LG] |
| (or arXiv:2611.02746v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02746 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.5281/zenodo.23906641
DOI(s) linking to related resources |
Submission history
From: Yiqiao Qiu [view email]
[v1]
Mon, 2 Nov 2026 08:35:18 UTC (565 KB)
[v2]
Tue, 3 Nov 2026 08:22:11 UTC (565 KB)
来源：arXiv cs.CL · arxiv.org