arXiv cs.CL· Zhuoyuan Hao, Zhuo Li, Wu Li, Fangming Liu, Min Zhang, Jing Li·· 11 小时前AI 评分32
研究揭示 LLM 推理中的 Echo of Prompt 机制，提出 ED-SFT 与 Echoic Prompting 方法
Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning
AI 导读
研究将大推理模型在思维链开头复述问题的现象命名为 Echo of Prompt（EOP），并将其形式化为拒绝采样下的概率成本。因果干预实验显示，向失败的 GSM8K 推理轨迹中重新插入 echo 可使两个推理模型的 exact match 提升 7.9-10.4 分，但对非推理基座模型无效。
正文 · 原文
Abstract:Recent work improves reasoning in large reasoning models (LRMs) by test-time scaling through self-consistency and parallel thinking, adding generic "thinking tokens", or presenting the question twice, as in re-reading (RE2). However, these approaches either inject task-agnostic tokens or mandate heuristics that do not explain, and often ignore, the spontaneous repetition that many LRMs exhibit at the head of their long chain-of-thought (CoT). In this work, we investigate the model's tendency to restate the question, which we dub Echo of Prompt (EOP), as a mechanism that shapes computation from the start and can be viewed as prompt repetition inside the reasoning trace rather than in the input. We formalize its probabilistic cost by casting echo removal as rejection sampling, and define the Echo Likelihood Gap as a computable proxy linking early repetition to likelihood gains and downstream accuracy. We find that (1) in a causal intervention, reinserting an echo into failed GSM8K traces raises exact match by 7.9-10.4 points for two reasoning models but not for a non-reasoning base model; and (2) likelihood analyses controlling for length and suffix indicate that EOP's benefits go beyond verbosity, while layer-wise attention studies show that EOP increases attention from answer tokens to the answer prefix in middle layers, consistent with an attention refocusing mechanism. Building on these findings, we develop Echo-Distilled SFT (ED-SFT) to instill echoing before reasoning via supervised fine-tuning, and training-free Echoic Prompting (EP) to ground the model on the prompt midway through the trace. We evaluate on GSM8K, MathQA, Hendrycks-MATH, AIME24, and MATH-500 under identical decoding settings and budgets, and observe gains over baselines, including the insertion of thinking tokens, in most settings. Our code is publicly released at this https URL.
| Comments: | ICLR 2026 (poster). Public on OpenReview since 2025-10-08: this https URL. Code: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2602.06600 [cs.CL] |
| (or arXiv:2602.06600v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2602.06600 arXiv-issued DOI via DataCite |
|
| Journal reference: | The Fourteenth International Conference on Learning Representations (ICLR 2026) |
Submission history
From: Zhuoyuan Hao [view email]
[v1]
Fri, 6 Feb 2026 10:53:26 UTC (2,175 KB)
[v2]
Wed, 28 Oct 2026 09:40:24 UTC (2,155 KB)
[v3]
Sat, 31 Oct 2026 07:11:35 UTC (2,155 KB)
来源：arXiv cs.CL · arxiv.org