arXiv cs.CL· Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, Yilin Cheng, Bichuan Feng, Pengfei Liu, Junfeng Fang, Xiang Wang·· 2 天前AI 评分24
SimCT：为跨分词器在线策略蒸馏恢复丢失的监督信号
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
AI 导读
SimCT 通过扩大监督空间解决跨分词器在线策略蒸馏中共享 token 匹配丢弃教师信号的问题：在共享 token 之外，比较教师与学生模型都能实现的短多 token 续写，且不改变 OPD 损失形式。
正文 · 原文
Abstract:On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose \textbf{\underline{Sim}ple \underline{C}ross-\underline{T}okenizer OPD (SimCT)}, which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at \href{this https URL}{this https URL}.
| Comments: | Accepted by NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.07711 [cs.CL] |
| (or arXiv:2605.07711v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07711 arXiv-issued DOI via DataCite |
Submission history
From: Jie Sun [view email]
[v1]
Fri, 8 May 2026 13:16:17 UTC (999 KB)
[v2]
Thu, 21 May 2026 08:28:27 UTC (1,002 KB)
[v3]
Thu, 15 Oct 2026 08:39:01 UTC (1,009 KB)
来源：arXiv cs.CL · arxiv.org