arXiv cs.AI· Yuhang Yang, Mingxuan Xia, Chao Ye, Shuai Zhu, Shouqing Yang, Haobo Wang, Shenzhi Yang, Zhengqing Zang, Chenglong Wang·· 3 天前AI 评分22
PreSO：面向在线策略自蒸馏的特权表示隔离方法
PreSO: Privileged Representation Isolation for On-Policy Self-Distillation
AI 导读
PreSO 通过紧凑提示限制特权信息中的题目细节，并从跨问题表示偏移中估计低秩子空间，将特权修正投影后叠加到学生表示上作为蒸馏目标。在五个竞赛级数学基准上，PreSO 在 Qwen3-1.7B、4B、8B 上平均准确率较基座模型分别提升 6.8、9.0、10.0 分，且各规模最终 checkpoint 平均成绩均为最高。
正文 · 原文
Abstract:On-policy self-distillation (OPSD) uses a self-teacher conditioned on privileged information to turn the sparse outcome signal of reinforcement learning with verifiable rewards (RLVR) into dense token-level supervision. Yet the teacher's advantage may depend on information unavailable to the student at inference, raising a central question: which privileged changes are worth distilling? We use cross-problem recurrence as an empirical cue for potentially reusable guidance. Comparing the representation shifts induced by full solutions and compact hints, we find that hint-induced shifts are more concentrated and directionally consistent across problems, and yield more sustained downstream gains despite revealing less information. Motivated by this association, we propose Privileged Representation Isolation for On-Policy Self-Distillation (PreSO). At the source, PreSO uses compact hints to limit the solution-specific detail entering the privileged shift. At the target, it estimates a leading low-rank subspace from shifts pooled across related problems, projects the privilege-induced correction onto this subspace, and adds it to the student's unprivileged representation to form the representation-level distillation target. The selected subspace serves as an operational proxy for recurring structure, rather than a guaranteed separation of transferable and instance-specific information. On five competition-level mathematics benchmarks, PreSO consistently outperforms strong baselines across Qwen3-1.7B, 4B, and 8B, improving average accuracy over the base model by 6.8, 9.0, and 10.0 points. Beyond peak accuracy, PreSO attains the highest final-checkpoint average at every model scale, with improvements that persist through the end of training.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14001 [cs.LG] |
| (or arXiv:2610.14001v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14001 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhang Yang [view email]
[v1]
Sun, 27 Sep 2026 08:48:42 UTC (1,446 KB)
来源：arXiv cs.AI · arxiv.org