arXiv cs.AI· Pratyay Dutta, Kowshik Thopalli, Vivek Narayanaswamy·· 10 天前AI 评分30
COMPASS：在语言模型中定位推理能力所在
COMPASS: Finding Where Reasoning Lives in Language Models
AI 导读
COMPASS 是一种推理时激活引导方法，利用模型自身直接作答的正确性信号定位可干预的注意力头，并沿正确性方向引导其激活。在三个模型家族和多个数学基准上，该方法平均将 GSM8K 准确率提升 16 个百分点，并以少 20-70% 的生成 token 接近 CoT 准确率。干预无需重新拟合即可迁移到未见基准，效果集中在极少数注意力头上。
正文 · 原文
Abstract:Explicitly eliciting reasoning substantially improves LLM performance. Existing approaches require a predefined characterization of reasoning, whether through CoT prompt design, contrastive CoT directions, or via SAE derived reasoning features. For mathematical reasoning with verifiable answers, we show that a much simpler signal suffices, which is the correctness of the model's own direct answer attempts. This signal yields a latent direction that elicits reasoning. This direction is decodable within the activations of most attention heads, but only a small subset of them can be effectively intervened. We introduce COMPASS, an inference-time steering method that identifies these heads using a logit-space attribution score and steers their activations along the correctness direction, requiring only per-head activation statistics. Across three model families and multiple math benchmarks, COMPASS outperforms the activation-steering baselines we compare against, improves GSM8K accuracy by 16 percentage points on average, and approaches CoT accuracy with 20-70\% fewer generated tokens. Interventions transfer without re-fitting to unseen benchmarks, and ablations show that both the correctness direction and the small set of heads carrying it are necessary, with the effect concentrated in remarkably few heads.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07469 [cs.AI] |
| (or arXiv:2610.07469v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07469 arXiv-issued DOI via DataCite |
Submission history
From: Pratyay Dutta [view email]
[v1]
Mon, 5 Oct 2026 22:35:47 UTC (213 KB)
[v2]
Wed, 7 Oct 2026 07:10:38 UTC (213 KB)
来源：arXiv cs.AI · arxiv.org