arXiv cs.CL· Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma·· 3 天前AI 评分23
FSPO：面向策略优化的函数空间时序自蒸馏
Function-Space Temporal Self-Distillation for Policy Optimization
AI 导读
研究者提出 Function-Space Policy Optimization（FSPO），一种单策略框架，将当前特权适应与函数空间时序整合相结合，通过存储稀疏的历史 next-token 分布作为固定时序目标，无需维护持久教师模型。
正文 · 原文
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget, improving over the strongest baseline by approximately 4% points on average while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.
| Comments: | Aryan Das, Vinay Kumar Verma: Equal Contributions. Under Review |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.19687 [cs.LG] |
| (or arXiv:2610.19687v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19687 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vinay Verma Kumar [view email]
[v1]
Wed, 14 Oct 2026 07:10:49 UTC (597 KB)
来源：arXiv cs.CL · arxiv.org