arXiv cs.CL· Fei Wu, Zhenrong Zhang, Jianshu Zhang, Jun Du·· 3 天前AI 评分24
VASPO：用特权价值估计实现自蒸馏的策略优化方法
VASPO: Self-Distillation with Privileged Value Estimation
AI 导读
VASPO 将特权自蒸馏信号与 PPO 结合，由 critic 基于预测的续写回报计算 token 级优势，自教师则通过有界概率比对选定的正优势进行调制。在六个数学推理基准上，VASPO 在 Qwen3-4B 和 OLMo3-7B 上平均准确率分别较 PPO 提升 2.59 和 1.81 个百分点，并在 Knights and Knaves 上取得所评估方法中的最高平均准确率。代码已开源。
正文 · 原文
Abstract:On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20516 [cs.CL] |
| (or arXiv:2610.20516v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20516 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fei Wu [view email]
[v1]
Wed, 14 Oct 2026 17:17:18 UTC (6,247 KB)
来源：arXiv cs.CL · arxiv.org