arXiv cs.CL· Zhimin Gao, Meibo Hu, Jiamian Wang, Zhiqiang Tao, Pichao Wang·· 12 小时前AI 评分31
PRDO：面向推理模型选择性预测的推理后分布优化
Beyond Outcome Rewards: Post-Rationale Distribution Optimization for Selective Prediction in Reasoning Models
AI 导读
研究者提出 Post-Rationale Distribution Optimization（PRDO），在每个采样推理链之后对答案分布施加 proper score，无需目标推理链或单独置信度标签。
正文 · 原文
Abstract:Reasoning models are trained to produce correct answers, yet selective deployment requires ranking which answers to accept or defer, and outcome-reward training does not directly optimize the post-rationale distribution used for that decision. We identify two mismatches: expected correctness is mode-seeking rather than distribution-eliciting, and scoring an average across rationales rewards disagreement unavailable to a single-rationale decision. We introduce Post-Rationale Distribution Optimization (PRDO), which applies a proper score to the answer distribution after each sampled rationale, without target rationales or separate confidence labels. Across four 0.6--3.8-billion-parameter models on MMLU-Pro, PRDO improves Brier score, error ranking and secondary accuracy relative to budget-matched, temperature-scaled GRPO. At 20\% realized error, oracle coverage rises by \npj{Q3a_vs_grpo.qwen3-0.6b.mp.oracle_cov@0.this http URL}--\npj{this http URL-4-mini.mp.oracle_cov@0.this http URL} percentage points, and ranking gains transfer to unseen tasks. Matched controls identify proper, per-rationale scoring as the active design choices, while verification tasks show the boundary: better selective ranking can coexist with lower full-coverage accuracy. Code is available at this https URL
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.38850 [cs.AI] |
| (or arXiv:2609.38850v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38850 arXiv-issued DOI via DataCite |
Submission history
From: Pichao Wang [view email]
[v1]
Wed, 30 Sep 2026 03:08:45 UTC (55 KB)
[v2]
Sun, 1 Nov 2026 19:28:31 UTC (545 KB)
来源：arXiv cs.CL · arxiv.org