arXiv cs.CL· Xiang Zhang, Lovedeep Gondara, Jekaterina Novikova·· 2 天前AI 评分39
当评审一致却出错：一致性-准确率解离成为无标签评估的盲区
When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation
AI 导读
研究直接检验了"一致性可代理正确性"这一无标签评估假设，使用四个金融问答数据集、十个一致性维度和两个前沿模型族，在单样本层面测量一致性与准确率的联合分布。结果显示，60% 的错误答案落在高一致性区域（一致性 ≥ 0.8），其中 77.7% 出现完全答案坍缩，即所有扰动变体给出相同错误答案。
正文 · 原文
Abstract:Label-free evaluation methods, such as self-consistency filtering, majority-vote aggregation, and agreement-based reliability scoring, rely on an implicit assumption that consistency can serve as a proxy for correctness when ground-truth labels are unavailable or expensive to obtain. In this paper, we test this assumption directly. Using four financial QA datasets (multiple-choice and free-form), ten consistency dimensions, and two frontier model families, we measure the joint distribution of consistency and accuracy at the level of individual examples rather than in aggregate. We find that 60% of all incorrect answers fall in the high-consistency region (consistency $\geq$ 0.8), and that 77.7% of these show complete answer collapse, i.e. identical wrong answers across all perturbation variants. This high-consistency/incorrect quadrant cannot be detected by any evaluation signal that is built on agreement alone, as the model appears maximally reliable while being wrong. We show that the phenomenon is robust to the choice of consistency threshold t (it persists from t=0.50 to t=1.00), is shared across model families, and is concentrated in specific intersections of perturbation dimension and domain. We map where this quadrant occurs, analyze its stability properties, and discuss the implications for evaluation pipelines that use agreement-based signals in place of human labels.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.21128 [cs.CL] |
| (or arXiv:2610.21128v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21128 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jekaterina Novikova Dr. [view email]
[v1]
Thu, 15 Oct 2026 02:33:30 UTC (927 KB)
来源：arXiv cs.CL · arxiv.org