arXiv cs.CL· Minghe Shen, Ananth Balashankar, Adam Fisch, David Madras, Miguel Rodrigues·· 10 小时前AI 评分31
面向有界劣质评判器的稳健 LLM 评估:约束最大似然方法
Robust LLM Evaluation with Boundedly Bad Judges: A Constrained Maximum Likelihood Approach
AI 导读
论文提出约束最大似然估计(CMLE),通过真实正例率与假正例率显式建模 LLM-as-a-judge 代理标签的误差,并用数据驱动的约束充当自适应正则化。在合成与真实 LLM 评估任务上,CMLE 的均方误差比非分层基线最多低约 9%,其分层变体 SCMLE 比最强基线最多低约 30%,低标签场景收益最大。
正文 · 原文
Abstract:Evaluating language models at scale often involves reconciling costly, but reliable, labels (e.g., from human raters) with cheap, but noisy, proxy labels (e.g., from LLM-as-a-judge autoraters). This paper proposes constrained maximum likelihood estimation (CMLE), a likelihood-based approach to large language model (LLM) evaluation that explicitly models the error in the proxy labels via their true-positive and false-positive rates, subject to data-driven constraints. Functionally, these constraints act as an adaptive regularization that restricts the feasible judge error rates, particularly when the available number of reliable labels is limited. In contrast to asymptotically unbiased estimators (such as those based on prediction-powered inference), this formulation accepts a controlled bias under misspecified constraints in exchange for variance reduction relative to various baselines including unconstrained maximum likelihood estimation (UMLE); we characterize this bias--variance tradeoff analytically under point constraints. We identify the regimes in which CMLE outperforms UMLE and prediction-powered baselines in mean squared error. We further extend the framework to heterogeneous settings using a stratified variant involving stratum-specific constraints based on judge confidence. Empirically, on synthetic and real-world LLM evaluation tasks, CMLE achieves up to roughly 9\% lower mean squared error (MSE) than non-stratified baselines, and its stratified variant (SCMLE) achieves up to roughly 30\% lower MSE than the strongest baseline considered, with the largest gains in low-label regimes.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.03257 [cs.CL] |
| (or arXiv:2604.03257v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.03257 arXiv-issued DOI via DataCite |
Submission history
From: Minghe Shen [view email]
[v1]
Wed, 11 Mar 2026 21:48:40 UTC (32,480 KB)
[v2]
Thu, 19 Nov 2026 16:14:08 UTC (36,553 KB)
来源:arXiv cs.CL · arxiv.org