arXiv cs.CL· Wenhang Shi, Yiren Chen, Zhe Zhao, Jinhao Dong, Wei Lu, Xiaoyong Du·· 2 天前AI 评分34
Self-Evolution Reasoner：通过策略内在探索与利用实现持续优化
Self-Evolution Reasoner: Continuous Optimization via Policy-Intrinsic Exploration and Exploitation
AI 导读
研究者提出 Self-Evolution Policy Optimization（SEPO）框架，用标签引导策略采样出正确推理，再将当前策略作为自教师把新解法蒸馏回模型，形成策略内在的探索与利用循环。在科学、工具使用和复杂逻辑推理任务上，相同训练时间或步数下 SEPO 均取得最佳表现，并在基线严重失效的复杂任务和小规模模型上仍保持稳定提升。代码已开源。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) enables large language models (LLMs) to continuously self-improve their reasoning capabilities. Compared to RLVR's scalar rewards, recent self-supervised on-policy distillation methods leverage denser supervision signals. However, both paradigms suffer from exploration bottlenecks, struggling to escape local optima when the policy fails to sample valid solutions. Relying on external demonstrations can bypass this issue; however, the off-policy nature hinders genuine self-improvement and the mismatched-distribution reasoning can not be fully exploited. In this work, we propose Self-Evolution Policy Optimization (SEPO), a framework that drives continuous reasoning capability evolution via guided exploration and in-distribution exploitation. Specifically, SEPO utilizes labels to elicit correct reasoning from the policy, unlocking the exploration potential. By treating current policy conditioned on this in-distribution reasoning as a self-teacher, SEPO distills the newly discovered solution back into the policy. Through this iterative cycle of policy-intrinsic exploration and exploitation, SEPO achieves robust and continuous capability evolution. Across scientific, tool-use, and complex logical reasoning domains, SEPO consistently achieves the best performance under identical training time or steps, demonstrating superior efficiency and convergence ceilings. Notably, SEPO maintains stable improvements even in scenarios where baselines severely underperform or fail entirely, such as on complex tasks or with small-scale models. Our code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34137 [cs.LG] |
| (or arXiv:2610.34137v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34137 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wenhang Shi [view email]
[v1]
Sun, 25 Oct 2026 08:18:39 UTC (308 KB)
来源：arXiv cs.CL · arxiv.org