arXiv cs.CL· Guofang Tang·· 10 小时前AI 评分38
研究：LLM 验证会让最小对基准对语言模型更简单
LLM Validation Makes Minimal-Pair Benchmarks Easier for Language Models
AI 导读
在 BLiMP 和 QFrBLiMP 上测试三种 LLM 验证方式后发现，强制选择验证器与留出人类多数意见的一致率高于五人评审团，但漏检人类拒绝标签的比例远高于单个评审。
正文 · 原文
Abstract:Minimal-pair benchmarks are only as good as their labels, and LLMs are an obvious substitute for costly human validation. We test it on BLiMP (English) and QFrBLiMP (Quebec French), the two public minimal-pair datasets releasing every forced-choice human judgment. Three LLMs validate by forced choice, by rating, and as a panel, scored against human panels of one, three and five judges in two pre-registered phases. On BLiMP, forced-choice validators agree with held-out human majorities more often than five-judge panels, yet catch far fewer pairs whose label humans reject than one judge does. By point estimate, no validator is worth more than one judge. The errors are not random. On both datasets, across 10 open LMs, human-supported pairs that the pre-registered LLM filter flags are 18.9 points harder than those it accepts, in all 70 LM-validator combinations, and harder than pairs a single human judge flags. Filtering thus removes items on which LMs fail but humans do not, raising accuracy more than random removal. A protocol frozen before any output replicates the gap and the lift on Dutch BLiMP-NL. LLM validation makes minimal-pair benchmarks easier for the models they are meant to test; we release code, validator outputs and item-level scores.
| Comments: | 11 pages, 1 figure |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.13788 [cs.CL] |
| (or arXiv:2611.13788v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13788 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tang Guofang [view email]
[v1]
Tue, 10 Nov 2026 09:22:38 UTC (110 KB)
来源：arXiv cs.CL · arxiv.org