arXiv cs.CL· Zijian Wu, Lingkai Kong, Lin Yang, Haiteng Zhao, Kuikun Liu, Kai Chen, Dahua Lin, Wenwei Zhang·· 11 小时前AI 评分38
Self-Steered Reasoner（SSR）：用可达性对齐的自蒸馏突破推理瓶颈
Steer, Don't Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation
AI 导读
研究者提出 Self-Steered Reasoner（SSR），通过学生支撑投影把特权教师分布转成仅覆盖无引导学生局部可达推理动作的方向性信号，并以验证筛选正确轨迹。在即使充分 RL 后 pass@64 仍为 0 的题目上，SSR 无需参数更新即为 74.6% 的题目恢复正确且可学习的轨迹；在这些轨迹上微调并恢复强化学习后，AIME25/26 与 HMMT26 聚合准确率提升 4.9 个百分点。
正文 · 原文
Abstract:Post training for large reasoning models remains insufficient on challenging problems, whose correct trajectories are hard to sample. Though prior works address this difficulty by integrating supervised fine tuning on expert trajectories with reinforcement learning, such external experts are not always available. Existing self improvement methods instead use reference solutions to create a privileged self-teacher. Yet privileged teacher correctness does not imply student reachability. Solution conditioning shifts the teacher distribution toward actions unlikely under the unprivileged student, producing seemingly "correct" trajectories that are unreachable or even mal-formed learning targets. Accordingly, we identify reachability alignment as the key to self constructed supervision and propose Self-Steered Reasoner (SSR). SSR realizes this principle through student support projection, which transforms the privileged teacher distribution into a directional signal only over reasoning actions locally reachable under the unguided student. Verification then retains correct trajectories sampled from this reachable space. To further boost the sampling efficiency, an iterative refinement framework is adopted, using diagnostic guidance based on current trajectories. On problems with pass@64 = 0 even after extensive RL, SSR recovers correct and learnable trajectories for 74.6% without parameter updates. Fine tuning on these trajectories followed by resumed reinforcement learning improves aggregate accuracy by 4.9 percentage points on AIME25/26 and HMMT26. These results show that privileged guidance becomes effective self-improvement data when aligned with the reasoning actions the student can reach.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.00796 [cs.CL] |
| (or arXiv:2611.00796v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00796 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zijian Wu [view email]
[v1]
Sat, 31 Oct 2026 10:18:31 UTC (719 KB)
来源：arXiv cs.CL · arxiv.org