JEV-as-a-Judge：置信时接受，不确定时升级

View PDF HTML (experimental)

Abstract:LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind where the verdict must be derived, as in math, code, and logic. Its confidence marks this boundary. With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee, and in a pre-specified live test on two new workloads the cascade matches GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose; we close with a simple recipe for validating thresholds locally.

Comments:	Expanded the dataset, updated the results and figures, and added new analyses. The previous result reporting 99% of GPT performance at 57% of the cost is retained in the appendix
Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2609.26550 [cs.AI]
	(or arXiv:2609.26550v4 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2609.26550 arXiv-issued DOI via DataCite

Submission history

From: Yubo Li [view email]
[v1] Tue, 22 Sep 2026 15:05:56 UTC (2,824 KB)
[v2] Sun, 27 Sep 2026 19:00:59 UTC (1,802 KB)
[v3] Tue, 29 Sep 2026 15:32:52 UTC (1,605 KB)
[v4] Tue, 6 Oct 2026 00:32:50 UTC (1,605 KB)