arXiv cs.AI· Kai Tang, Xing Zheng, Haobo Wang, Guangcheng Zhu, Shenzhi Yang, Zhengqing Zang, Yingfan Ma, Bowen Song, Weiqiang Wang·· 12 小时前AI 评分27
SimOPD：通过响应相似度实现 on-policy 蒸馏的关系型知识迁移
SimOPD: Relational Knowledge Transfer for On-Policy Distillation
AI 导读
SimOPD 提出用响应相似度迁移教师模型内部知识，让学生模型在自身表示空间内匹配教师给出的相似度分布，无需跨模型投影。在同架构与跨架构设置下，其整体平均成绩分别比最强基线最高提升 2.6 和 3.1 个百分点。该方法使用学生生成响应的成对记忆库提取监督信号，避免直接隐状态匹配带来的特征空间偏移问题。
正文 · 原文
Abstract:A key challenge in on-policy distillation (OPD) is extracting effective supervision from teacher knowledge. Supervision from token probabilities may underutilize internal knowledge, while direct hidden-state matching forces the student to shift to the teacher's feature space and complicates cross-architecture transfer. Inspired by contrastive learning's use of relationships between instances, we propose \textbf{SimOPD} to transfer teacher internal knowledge through response similarities. Specifically, SimOPD uses paired memory banks of student-generated responses to extract supervision from the teacher's internal representations as similarity distributions. The student then matches these distributions within its own representation space, retaining representational freedom and removing the need for cross-model projection. Extensive experiments show that SimOPD improves the overall average over the strongest baselines by up to 2.6 and 3.1 percentage points in same- and cross-architecture settings.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.23306 [cs.AI] |
| (or arXiv:2610.23306v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23306 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kai Tang [view email]
[v1]
Fri, 16 Oct 2026 11:10:21 UTC (453 KB)
来源：arXiv cs.AI · arxiv.org