arXiv cs.CL· Niall Dalton·· 10 小时前AI 评分29
后训练如何影响语言模型的口头算法计算能力
How Post-Training Influences Verbalized Algorithmic Computation in Language Models
AI 导读
研究追踪 Olmo-3-7B-Instruct 在连续 SFT、DPO、RLVR 检查点上的多位数加法表现，发现 SFT 与 RLVR 的探针从内部激活中恢复进位信息的准确率相近，但操纵该信息无法可靠控制模型输出。
正文 · 原文
Abstract:Post-training improves language-model reasoning, but how it changes the use of internal information and verbalized chain of thought (CoT) remains unclear. We study successive SFT, DPO, and RLVR checkpoints of Olmo-3-7B-Instruct on multi-digit addition, using carry values as known intermediate state. Probes recover carry information from internal activations with similar accuracy in SFT and RLVR, although manipulating this information does not reliably control model outputs. In contrast, the usefulness of verbalized reasoning differs: CoTs produced by RLVR yield higher accuracy across receiving checkpoints than those produced by SFT or DPO. We also show that carry-related statements receive higher importance scores after RLVR than matched random controls. DPO shows similar sensitivity to carry-related statements but without comparable accuracy gains. These results suggest that post-training changes the usefulness of written reasoning and sensitivity to its content in meaningful ways, although greater sensitivity does not necessarily coincide with better problem performance.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.13552 [cs.CL] |
| (or arXiv:2611.13552v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13552 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Niall Dalton [view email]
[v1]
Tue, 10 Nov 2026 04:12:09 UTC (293 KB)
来源：arXiv cs.CL · arxiv.org