arXiv cs.CL· Zefeng He, Yichen He, Yuan Lin, Yang Yang, Yu Cheng·· 13 小时前AI 评分34
RLEC：让 LLM 按显式置信目标学会回答或弃答的强化学习方法
Reinforcement Learning Done Right for Explicit Confidence Targets
AI 导读
研究者提出 RLEC（Reinforcement Learning with Explicit Confidence Targets），首次成功用单一 LLM 实现显式置信目标控制——用户在提示词中指定目标 p，模型自行决定回答或弃答，使已回答问题的准确率至少达到 p。
正文 · 原文
Abstract:Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliability. Training models to abstain with reinforcement learning is a natural approach, but most existing methods use a fixed error penalty and therefore cannot adapt their behavior to different confidence requirements at inference time. We study \emph{explicit confidence targets}, where a user specifies a target \(p\) in the prompt and the model directly chooses whether to answer or abstain, with the goal of achieving an accuracy of at least \(p\) among answered questions. However, learning to respond to different confidence targets with a single model has proven difficult, and prior attempts have failed to distinguish different targets. We show that this failure comes from two problems: the reward signal that distinguishes different values of $p$ is often lost during optimization, while the original error penalty grows without bound as $p$ approaches one. We propose RLEC (\textbf{R}einforcement \textbf{L}earning with \textbf{E}xplicit \textbf{C}onfidence Targets), which preserves the learning signal needed to distinguish confidence targets, introduces a stable reward family that keeps the desired answer condition unchanged, and uses reward scheduling to balance effective learning and training stability. Together, these designs enable the first successful training of a single LLM for explicit confidence target control. Extensive experiments across diverse benchmarks and model scales show that RLEC largely preserves general capabilities and achieves competitive calibration performance. RLEC also supports fine-grained target control without an external decision system or threshold calibration after training. By making explicit confidence targets part of the trained policy, RLEC opens a new and more direct path toward behavioral calibration.
| Comments: | 22 pages, 13 figures |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35059 [cs.LG] |
| (or arXiv:2610.35059v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35059 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zefeng He [view email]
[v1]
Mon, 26 Oct 2026 09:05:29 UTC (1,063 KB)
来源：arXiv cs.CL · arxiv.org