arXiv cs.AI· Gukhyeon Lee, SangKeun Lee·· 9 天前AI 评分24
RL-ARC：通过推理引导的不确定性校准大型推理模型
RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty
AI 导读
针对 RLVR 训练导致推理模型校准退化、过度自信的问题，研究者提出校准感知训练框架 RL-ARC，联合利用推理置信度与答案置信度，将推理置信度作为正确样本的推理引导正则项、错误样本的过度自信惩罚项。在 ID 与 OOD 设置下，RL-ARC 在改善校准的同时让推理模型能根据问题自适应估计置信度，且未显著牺牲推理性能。
正文 · 原文
Abstract:Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.
| Comments: | AACL-IJCNLP 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.11352 [cs.AI] |
| (or arXiv:2610.11352v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11352 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Gukhyeon Lee [view email]
[v1]
Thu, 8 Oct 2026 06:45:24 UTC (4,638 KB)
来源：arXiv cs.AI · arxiv.org