智能体 AI 评测的有效性危机：三层复合效度模型揭示基准分数可靠性衰减

View PDF HTML (experimental)

Abstract:Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims. No formal framework has characterized how validity degrades across these stages. We present a knowledge-based three-layer compounding validity model, $V_{\rm total} \leq V_1 \times V_2 \times V_3$, that captures multiplicative degradation across task generation ($V_1$), human-simulator calibration ($V_2$), and automated judgment ($V_3$). Applying it to five widely-used agentic benchmark configurations, we find $V_1$ and $V_2$ independently measurable and both degraded, but $V_3$ not currently quantifiable for any: three of the four benchmarks score with a single deterministic rule, so no second rater exists and reliability is undefined, and the fourth's partial LLM grader has no published grader-vs-human measurement.
A structured survey of 55 published agentic evaluation papers finds that 26 of them report no chance-corrected inter-rater statistic of any kind, a determination four independent raters recover at Krippendorff's $\alpha = 0.834$. On the broader judgment of whether a reported metric fits its rating design, approximately 80% of this purposive sample use structurally mismatched, incomplete, or absent inter-rater reliability (IRR) metrics (four-rater range 60.7%-84.4%). Both are consistent with systematic $V_3$ collapse. We further document $V_1$ failures (task validity flaws in 7 of 10 benchmarks) and $V_2$ miscalibration (up to 9 percentage points inter-simulator variance, with systematic demographic disparities for non-Standard American English speakers).
We derive eight prescriptions grounded in psychometric science and domain-stratified reliability thresholds (intraclass correlation $\geq 0.70$; $\alpha \geq 0.67/0.70/0.80$ by consequence level) that practitioners can apply immediately.

Comments:	50 pages, 2 figures, 7 tables. Second revision (R2) under review at Knowledge-Based Systems (Elsevier). Code and data: this https URL
Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2608.00794 [cs.AI]
	(or arXiv:2608.00794v5 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2608.00794 arXiv-issued DOI via DataCite

Submission history

From: William Caban [view email]
[v1] Sat, 1 Aug 2026 17:50:12 UTC (45 KB)
[v2] Tue, 4 Aug 2026 16:49:21 UTC (33 KB)
[v3] Wed, 5 Aug 2026 05:37:25 UTC (33 KB)
[v4] Fri, 11 Sep 2026 00:32:02 UTC (35 KB)
[v5] Mon, 26 Oct 2026 14:57:54 UTC (61 KB)