arXiv cs.AI· Taehyeon Kim, Hyunsoo Lee, Youngsoo Jang, Moontae Lee·· 3 天前AI 评分27
RPO：通过根 token 策略优化让大推理模型自适应思考
Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking
AI 导读
研究提出 RPO（Root-token Policy Optimization），通过组相对奖励和组内优势仅训练初始根 token（如是否调用思考标签），让大推理模型自行决定何时推理。该方法在多个模型族和规模上实现难度感知的自适应思考，训练算力仅为先前自适应推理方法的约 2%，并大幅降低显存占用。该工作已被 ACL 2026 接收为 Oral。
正文 · 原文
Abstract:Large reasoning models (LRMs) achieve strong performance by externalizing explicit reasoning traces before producing the answer, yet suffer from overthinking challenge that allocates uniformly heavy computation to queries of varying difficulty. While proprietary models mitigate this via opaque routing, open-source LRMs still lack an efficient mechanism to internalize adaptive reasoning due to both expensive training cost and limited disclosure of training recipes. In response, we introduce RPO (Root-token Policy Optimization), a framework that enables LRMs to self-determine when to reason by training only the initial root token (e.g., whether to invoke the <think> tag) via group relative reward and group-wise advantages. By focusing on this pivotal branching point, RPO drastically reduces training overhead and VRAM usage. Across multiple model families and scales, RPO learns difficulty-aware adaptive thinking at just ~2% of the training compute of prior adaptive-reasoning methods.
| Comments: | ACL 2026 Oral |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.17767 [cs.CL] |
| (or arXiv:2610.17767v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17767 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Taehyeon Kim [view email]
[v1]
Tue, 13 Oct 2026 05:53:34 UTC (474 KB)
来源：arXiv cs.AI · arxiv.org