arXiv cs.CL· Utkarsh Bahuguna·· 13 小时前AI 评分39
多数投票反而拖累小模型：自一致性在多数 GPQA Diamond 难题上降低准确率
When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMs
AI 导读
多数投票式自一致性（SC）在小型指令微调模型上反而降低了大多数 GPQA Diamond 问题的单题准确率，Qwen2.5-7B 有 56.6% 的问题变差，Llama-3-8B 为 65.7%。
正文 · 原文
Abstract:Majority-vote self-consistency (SC) lowers per-problem accuracy on most GPQA Diamond problems for small instruction-tuned models: on 56.6% of problems for Qwen2.5-7B and 65.7% for Llama-3-8B. An obvious fix is a verifier-free confidence gate that decides which problems to vote on. We report that the most natural version of that fix also fails, and we separate three signal failures that v1 treated as one. The token-entropy gate fails for a measurement reason: averaged over a chain of 602 tokens, entropy mostly measures fluency. The chain average sits 0.0005 nats from the non-answer average and 0.1695 nats from the answer token. So we measure confidence at the answer token itself. On Qwen2.5-7B-Instruct-Turbo (198 problems, 64 samples each), a sample that disagrees with its problem's plurality still emits its answer at a median margin of at least 20.4 nats, and at least 75.5% of such samples exceed 10 nats. Both quantities were pre-registered and tested once on 69 problems that no exploratory analysis had seen; both passed. The unit matters. Pooled across the benchmark, the margin does separate correct from incorrect samples: the fraction above 10 nats differs by +0.0586 [+0.0160, +0.0994]. Within a problem, paired, it does not: -0.0142 [-0.0501, +0.0215]. Token log-probabilities do carry information, but not for the within-question choice that self-consistency makes. The agreement gate's failure has no mechanism we can identify, and we leave it open. The new claims rest on one model: a registered replication on a second model was sampled but could not be evaluated, and we report that instead. Separately, three reasoning models could not be evaluated on hosted serverless inference at a small budget, each for a different measured reason. All three are downloadable, so this is a limit of metered APIs, not of what can be known.
| Comments: | 23 pages, 5 figures, 4 tables. v3: corrections and disclosures. Section 9 verdicts marked unsupported; Section 4.2 oracle, 4.3 captures and 4.5 primaries restated; MD1/MD2 reported as bounds. No new experiments |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.11403 [cs.AI] |
| (or arXiv:2608.11403v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.11403 arXiv-issued DOI via DataCite |
Submission history
From: Utkarsh Bahuguna [view email]
[v1]
Tue, 11 Aug 2026 20:08:51 UTC (340 KB)
[v2]
Sat, 15 Aug 2026 22:30:00 UTC (425 KB)
[v3]
Wed, 21 Oct 2026 18:18:04 UTC (430 KB)
来源：arXiv cs.CL · arxiv.org