从自置信到统计精度：用直接审计与 LLM 辅助审计扩展人在回路评估
From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing
该论文提出统一统计框架，用于在部分人工核验下评估 LLM 与 MLLM：直接审计路线采用置信度分层抽样，无需校准置信度即可还原总体准确率；LLM-as-a-judge 路线则以敏感度和假阳性率建模，给出带校准不确定性的误差校正估计。
Authors:Harine Choi (1), Chenchen Kuai (2), Zihao Li (3), Jiwan Jiang (2), Yunlong Zhang (2), Zhiwen Fan (4), Zhengzhong Tu (5), Yang Zhou (2) ((1) Texas A&M Transportation Institute, Texas A&M University System, (2) Zachry Department of Civil and Environmental Engineering, Texas A&M University, (3) Department of Civil, Construction and Environmental Engineering, Marquette University, (4) Department of Electrical and Computer Engineering, Texas A&M University, (5) Department of Computer Science and Engineering, Texas A&M University)
Abstract:For large language models (LLMs) and multimodal large language models (MLLMs), model-reported self-confidence is often miscalibrated, yet it may still be useful for efficient evaluation. At the same time, LLM-as-a-judge offers a scalable alternative to direct human auditing, but its practical benefit depends on calibration quality and inference cost.
This paper develops a unified statistical framework for both evaluation routes under alternative sampling strategies. For direct auditing, confidence-stratified sampling is used to recover population accuracy from partially verified outputs without requiring calibrated confidence. Finite-sample uncertainty is characterized, and the analysis shows how highly uneven confidence strata affect audit efficiency and label allocation. For judge-assisted evaluation, the judge is modeled through sensitivity and false-positive rate, yielding an error-corrected estimator with calibration uncertainty. The two routes are then compared through human-label and total-cost efficiency boundaries.
Experiments on MMLU-Pro and MMMU-Pro show that miscalibrated self-confidence can still support statistically valid stratified auditing. Under highly imbalanced confidence distributions, proportional allocation substantially reduces estimation error relative to equal-per-stratum allocation. For judge-assisted evaluation, only a small subset of target--verifier pairs satisfies the idealized human-label efficiency condition, and automated inference costs can further limit potential savings. These findings clarify the statistical and economic conditions under which confidence-guided auditing and automated verification are useful.
| Comments: | 21 pages, 8 figures. Reproduction artifacts available at this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35698 [cs.CL] |
| (or arXiv:2610.35698v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35698 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Harine Choi [view email]
[v1]
Mon, 26 Oct 2026 16:25:11 UTC (592 KB)
来源：arXiv cs.CL · arxiv.org