arXiv cs.CL· Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin·· 12 小时前AI 评分44
多种偏见共用同一回路：后训练调优预训练模型的线索路由机制
Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism
AI 导读
语言模型在提示词含错误线索（如“我认为答案是 B”）时会放弃正确答案，研究发现多种线索类型在多数模型上依赖预训练阶段就已存在的同一组注意力头。仅用约 200 条无线索的通用助手样例微调基座模型，即可在对话格式下激活这些头，说明跟随线索可能是助手微调的副作用；在 Tulu-3-8B 和 OLMo-2-7B 中交换六个中间层的注意力模式，可将建议答案线索的影响大部分转移至另一检查点水平。
正文 · 原文
Abstract:Language models often abandon a correct answer when the prompt contains a cue, a hint toward a wrong option such as "I think the answer is B." Sycophancy and related biases show this failure. They are often studied separately and blamed on alignment. We test this view causally, tracing the attention heads behind these biases through the stages of two post-training pipelines. In multiple-choice tests on most models, four cue types rely largely on the same few heads, a shared circuit already present in pretrained models. Most base models follow cues in plain text. Fine-tuning a base model on about 200 generic assistant examples without cues activates these heads in chat format, so cue-following can be a side effect of assistant tuning. The final reinforcement learning (RL) stage strengthens the effect of cues in almost every comparison. In Tulu-3-8B and OLMo-2-7B, we swap the attention patterns of six middle layers between the RL checkpoint and its predecessor. For suggested-answer cues, this shifts each checkpoint's cue effect most of the way to the other's level. Post-training thus seems to change when and how strongly models use this circuit. Mitigation for these multiple-choice biases could target this circuit.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.22937 [cs.CL] |
| (or arXiv:2610.22937v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22937 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Prakhar Gupta [view email]
[v1]
Fri, 16 Oct 2026 05:00:12 UTC (166 KB)
来源：arXiv cs.CL · arxiv.org