arXiv cs.CL· Stepan Epifantsev, Anna Antipova, Oleg Chernikov, Leon Strelkov, Alexey Dontsov·· 11 小时前AI 评分42
强制输出选项字母会抑制拒答：WMDP 上的机制性证据
Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDP
AI 导读
研究在 WMDP 数据集上对 Llama-3.1-8B 和 Qwen3-8B 给出多选题格式降低拒答率的首个机制性解释：模型处理 MCQ 时会迅速锁定答案字母 token，其激活距拒答方向约为开放式格式的 3 倍。消融承载 MCQ 答案的注意力头会提升拒答率，说明答案选择与拒答相互竞争。作者据此认为 MCQ 安全分数主要衡量知识能力，而非安全机制。
正文 · 原文
Abstract:Safety benchmarks for large language models (LLMs) commonly use multiple-choice question (MCQ) format. However, prior studies documented that MCQ format reduces refusal rates, which may significantly affect the safety behavior of the model. We provide a first mechanistic explanation of this effect on the Weapons of Mass Destruction Proxy (WMDP) dataset across two models - Llama-3.1-8B and Qwen3-8B. First, we use the tuned lens technique to show that when processing MCQ the model commits to an answer-letter token very quickly, which is in contrast to the gradual refusal build-up in open-ended formats. Additionally, we observe that activations on the MCQ format lie roughly 3x further from the refusal direction, compared with their open-ended counterparts. Moreover, ablating the attention heads that carry the MCQ format answer increases the refusal, which is a direct evidence that the MCQ answer-selection competes with refusal. These findings indicate that MCQ safety scores primarily measure knowledge capabilities and do not assess the safety-mechanisms.
| Comments: | Accepted to BlackboxNLP 2026 (EMNLP 2026 workshop). 17 pages, 15 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.08639 [cs.CL] |
| (or arXiv:2611.08639v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08639 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Stepan Epifantsev [view email]
[v1]
Thu, 5 Nov 2026 19:36:44 UTC (4,997 KB)
来源：arXiv cs.CL · arxiv.org