arXiv cs.AI· Minju Gwak, Alan Ritter·· 1 天前AI 评分31
辅助奖励驱动的自我改进：从已验证推理轨迹中学习并评估迁移效果
Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer
AI 导读
研究提出用辅助轨迹偏好区分结果同样正确的推理轨迹，以提升小模型在有限监督下的自我改进与迁移能力。在线实验中，基于熵的 UID-Conj. 指标在 Qwen3-1.7B 上于 30 道无参考答案的 AIME 题上适配。
正文 · 原文
Abstract:Answer-level rewards cannot distinguish reasoning traces with equally correct outcomes. We study whether auxiliary trajectory preferences improve small-language-model self-improvement and transfer under limited supervision. We compare confidence-based (SC) and entropy-based metrics (UID-Conj.), which combines global entropy variation and local smoothness, through online reward augmentation and offline training on correct responses. Online experiments adapt Qwen3-1.7B on 30 AIME problems without reference answers; offline experiments use historical AIME problems. UID-Conj. improves the online macro-average across the adaptation benchmark and three transfer benchmarks by 2.08 percentage points over unmodified TTRL. The strongest evidence comes from offline weighted SFT: UID-Conj. exceeds uniform weighting on the same response pool by 1.93 points, with a paired prompt-bootstrap 95\% CI of [0.44, 3.44]. Benefits are task-dependent: SC rewards judged trace quality and instruction following, while UID-Conj. rewards yield stronger training outcomes in several settings. We find that auxiliary preferences can expose learning value among equally correct traces not captured during inference time, motivating their evaluation through downstream gains.
| Comments: | 28 pages, Work in Progress |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41296 [cs.AI] |
| (or arXiv:2610.41296v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41296 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Minju Gwak [view email]
[v1]
Fri, 30 Oct 2026 05:57:57 UTC (1,874 KB)
来源：arXiv cs.AI · arxiv.org