arXiv cs.AI· Izhar Ali·· 12 小时前AI 评分29
LLM 错误中的跨问题结构从何而来：共享随机种子、共享提示词与答案字母偏好
Where Cross-Question Structure in LLM Errors Comes From: Shared Seeds, Shared Prompts, and Answer-Letter Preference
AI 导读
研究用问题相关矩阵特征值检验 LLM 错误结构，发现单模型内独立作答无可检测结构，而同一轮基准测试共享随机种子或少样本提示词时结构才出现。跨 24 个模型，超出整体能力的最大结构是对答案字母的偏好：旋转选项后偏好跟随字母而非题目，且主要来自 Llama-3.2-1B，剔除后仅剩较小超额。
正文 · 原文
Abstract:Disagreement between language models is often read as missing knowledge. Questions that need the same missing knowledge should then be right or wrong together. We test for this structure with the eigenvalues of the questions' correlation matrix. Within one model, independently drawn answers show no detectable structure; it appears when all questions in one pass over the benchmark share a random seed or few-shot prompt. Across 24 models, the largest structure beyond overall skill is a preference among answer letters: when the options are rotated, it follows the letter, not the question, and it depends on one model, Llama-3.2-1B; without it only a smaller excess remains. Counting a question correct only if right in every rotation removes the part that acts on guesses. What remains is a marginal excess at the 5% line, tied post hoc to two near-duplicate model pairs and to Llama-3.2-1B.
| Comments: | 12 pages, 3 figures, 3 tables. Accepted at EIML@ICML 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.20464 [cs.AI] |
| (or arXiv:2607.20464v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.20464 arXiv-issued DOI via DataCite |
Submission history
From: Izhar Ali [view email]
[v1]
Mon, 18 May 2026 14:40:37 UTC (179 KB)
[v2]
Fri, 23 Oct 2026 02:03:05 UTC (762 KB)
来源：arXiv cs.AI · arxiv.org