arXiv cs.AI· Haiquan Hu, Yuzhu Liang, Weicheng Tang, Yanzeng Li, Yao Shi, Li Xu, Tian Wang·· 3 天前AI 评分22
BSDProbe：用条目级能力边界诊断 LLM 基准可靠性
Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries
AI 导读
BSDProbe 提出条目级基准结构诊断框架，通过沿经验排序的模型轴重复采样，构建边界位置、边界宽度、边界信号有效性与顺序一致性四维画像，并在六个基准上识别出不同比例的可疑条目。以 Qwen3 同时作为选择轴与参考轴时，其归一化区分度比各基准最佳基线平均相对提升 17.9%，与完整基准的排序一致性也达到或超过所有基线。
正文 · 原文
Abstract:Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their intended measurement purpose. BSDProbe provides an item-level framework for benchmark structural diagnosis that jointly characterizes correctness transitions and their supporting response evidence. Repeated responses along an empirically ordered model axis yield complementary correctness and normalized entropy trajectories, from which four-dimensional profiles of boundary position, boundary width, boundary-signal validity, and order consistency are constructed for benchmark-level aggregation. Experiments on six benchmarks identify varying proportions of potentially anomalous items. With Qwen3 as both the selection and reference axis, BSDProbe achieves a 17.9\% mean relative improvement in normalized discrimination ratio over the best baseline per benchmark and matches or exceeds all baselines in ranking agreement with the full benchmark.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.33196 [cs.AI] |
| (or arXiv:2609.33196v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.33196 arXiv-issued DOI via DataCite |
Submission history
From: Haiquan Hu [view email]
[v1]
Sun, 27 Sep 2026 04:30:18 UTC (481 KB)
[v2]
Mon, 12 Oct 2026 13:48:38 UTC (1,791 KB)
来源：arXiv cs.AI · arxiv.org