arXiv cs.MA· Alessandro Rozza, Davide Mambelli, Steven Motta, Juri Taresch·· 2 天前AI 评分26
语言模型集成如何无标签估计可靠性：陪审团而非选民
A Jury, Not an Electorate: Label-Free Reliability Estimation for Ensembles of Language Models
AI 导读
研究者提出将语言模型集成视为陪审团，通过专家间一致与分歧模式，用EM算法在无标签条件下估计各专家可靠性，其决策规则等价于对专家答案分布的log-odds加权线性评分。在1560题基准上，9个最大3B参数的专家集成表现与使用标签拟合权重的融合规则相当，监督未带来统计显著优势，同时优于匹配自洽性的最佳单模型和最强的无标签替代方案。
正文 · 原文
Abstract:An ensemble of language models is a jury, not an electorate. The answers are testimony from witnesses of unknown reliability, and the question is not what the majority said but who is credible. Most ensemble methods assign each expert a fixed reputation, often learned from labels. We ask whether it can instead be inferred from the pattern of agreement and disagreement among the experts. We model the experts as a jury: each question has a latent true answer, each expert reports it with unknown reliability, and those reliabilities are estimated by Expectation Maximisation (EM) from unlabelled answer configurations alone. We show that its decision rule reduces to a log-odds weighted linear score over the experts' answer distributions. What changes is not how the jury votes, but how it learns whom to trust. On a 1560-question benchmark, the jury, with nine experts of up to 3B parameters, performs comparably to the same fusion rule with label-fitted weights, showing no statistically significant advantage from supervision. Instead, it outperforms the best single model with matched self-consistency and the strongest label-free alternative on the full benchmark.
| Comments: | 35 pages, 4 figures, 13 tables, including appendices |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2610.21085 [cs.CL] |
| (or arXiv:2610.21085v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21085 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Steven Motta [view email]
[v1]
Thu, 15 Oct 2026 01:40:04 UTC (152 KB)
来源：arXiv cs.MA · arxiv.org