arXiv cs.CL· Zongye Lyu·· 3 天前AI 评分44
同一答案换个框：RLVR 的域外增益有多少来自答案格式？
Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?
AI 导读
研究测量了 RLVR 报告的域外增益中有多少来自答案格式而非答案本身。在四个已发布的 7B SimpleRL base-RL 模型对上，用公开代码提取器报告的域外多选题增益有 82% 在改用能同时读取 \boxed{} 与正文的提取器后消失；独立 LLM 读者给出的比例为 77%，要求输出框的提示词下为 79%。而在另一个配方训练的 32B 模型对上，报告的增益几乎不含格式增益。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) is credited with gains beyond the domain its rewards check, such as higher MMLU accuracy after training on mathematics. Such accuracies are usually read out of the model's text by an answer extractor. A model that learns to write its answer where the extractor looks can therefore score higher without answering better. We measure how much of the gain reported by extractors copied from public code arises this way. We generate outputs from a base model and its RL model and score them twice: with those extractors, and with one extractor of our own that reads a stated answer whether it is in the answer box \boxed{} or in the text. On four released 7B SimpleRL base-RL pairs, 82% of the out-of-domain multiple-choice gain that these extractors report disappears under ours: it is a gain in answer format, not in answers. The base models already state most of their answers in text, and RL moves them into the box, where these extractors look first. Independent LLM readers, in place of our extractor, put the figure at 77%. With a prompt that itself asks for the box, our extractor puts it at 79%. On a 32B pair trained with another recipe, almost none of the reported gain is format gain: its base model boxes its answers when asked, and its RL model rarely does when not. A reported gain is partly a property of the extractor that read it, and re-scoring stored outputs with a second extractor shows how much, without re-running RL.
| Comments: | 36 pages (10 main), 5 figures, 28 tables |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2610.20639 [cs.LG] |
| (or arXiv:2610.20639v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20639 arXiv-issued DOI via DataCite |
Submission history
From: Zongye Lyu [view email]
[v1]
Fri, 2 Oct 2026 12:33:17 UTC (3,633 KB)
来源：arXiv cs.CL · arxiv.org