arXiv cs.AI· Karl Hanna, Chen Feng·· 3 天前AI 评分27
LLM 多选题评测中准确率与顺序敏感性为何会背离
A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation
AI 导读
研究用两个闭源模型和两个开源权重模型在 MMLU 与 ARC-Challenge 上测试四类策略，发现准确率与顺序敏感性并不一致变化：两阶段提示词在 MMLU 上降低了 GPT 的翻转率和 RStd，准确率却无显著变化。
正文 · 原文
Abstract:Multiple-choice question (MCQ) benchmarks are widely used to evaluate large language models, yet a single reported accuracy number is not enough to distinguish a model's latent ability from order sensitivity. Debiasing methods tend to be judged on one axis, which implicitly assumes that fixing order sensitivity and improving accuracy go together. We test this assumption with two proprietary models and two open-weight models on MMLU and ARC-Challenge, across four families of strategies. We measure order sensitivity with flip rate, capturing per-question instability, and recall standard deviation (RStd), capturing aggregate positional imbalance. We find accuracy and order sensitivity do not move together consistently: two-stage prompting reduces both flip rate and RStd for GPT on MMLU without a significant change in accuracy. The two sensitivity metrics also diverge from each other. Adding reasoning to direct answering leaves Llama's MMLU flip rate statistically indistinguishable while RStd falls sharply. A $2\times2$ ablation across generation-then-matching, crossing option visibility with matcher type, shows that withholding options mainly harms semantic matching, which achieves zero flip rate by construction, yet loses significantly in accuracy; the effect is smaller when using an LLM matcher. We recommend reporting flip rate and RStd along with accuracy when evaluating MCQ strategies.
| Comments: | Submitted to ARR, aimed at NAACL2027. Code available at this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.11947 [cs.CL] |
| (or arXiv:2608.11947v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.11947 arXiv-issued DOI via DataCite |
Submission history
From: Karl Hanna [view email]
[v1]
Wed, 12 Aug 2026 11:34:45 UTC (77 KB)
[v2]
Sat, 10 Oct 2026 23:56:33 UTC (79 KB)
来源：arXiv cs.AI · arxiv.org