AnytimeReasoner：用 Budget Relative Policy Optimization 优化任意 token 预算下的推理

View PDF HTML (experimental)

Abstract:Scaling test-time compute is crucial for enhancing the reasoning capabilities of large language models (LLMs). Existing approaches typically employ reinforcement learning (RL) to maximize a verifiable reward obtained at the end of reasoning traces. However, such methods optimize only the final performance under a large and fixed token budget, which hinders efficiency in both training and deployment. In this work, we present a novel framework, AnytimeReasoner, to optimize anytime reasoning performance, which aims to improve token efficiency and the flexibility of reasoning under varying token budget constraints. To achieve this, we truncate the complete thinking process to fit within sampled token budgets from a prior distribution, compelling the model to summarize the optimal answer for each truncated thinking for verification. This introduces verifiable dense rewards into the reasoning process, facilitating more effective credit assignment in RL optimization. We then optimize the thinking and summary policies in a decoupled manner to maximize the cumulative reward. Additionally, we introduce a novel variance reduction technique, Budget Relative Policy Optimization (BRPO), to enhance the robustness and efficiency of the learning process when reinforcing the thinking policy. Empirical results in mathematical reasoning tasks demonstrate that our method consistently outperforms GRPO across all thinking budgets under various prior distributions, enhancing both training and token efficiency.

Submission history

From: Penghui Qi [view email]
[v1] Mon, 19 May 2025 17:58:44 UTC (1,439 KB)
[v2] Thu, 5 Jun 2025 13:55:06 UTC (1,445 KB)
[v3] Fri, 7 Nov 2025 07:01:30 UTC (1,508 KB)
[v4] Fri, 30 Oct 2026 15:06:09 UTC (1,509 KB)