arXiv cs.CY· Russell Yang, Ruishi Chen, Pierce Kelaita, Riya Ranjan, Sibo Ma, Charles Dickens, Matthew Guillod, Megan Ma, Julian Nyarko·· 11 小时前AI 评分45
JudgmentBench：对比评分标准与偏好评估在质量评测中的差异
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
AI 导读
研究者发布 JudgmentBench，包含 30 个真实法律任务、1539 条评分标准打分与 1530 条成对偏好判断，均由执业律师标注，是首个在同一批专家、同一批任务上同时采集两类监督信号的高专业领域公开数据集。
正文 · 原文
Abstract:Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench, a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys--including at major U.S. law firms--with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same pool of experts on the same tasks. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics under both a per-task rank-correlation metric (mean Spearman's rank correlation of 0.908 vs. 0.150, estimated difference = 0.758 [0.494, 1.021]) and a per-judgment pairwise win-rate metric (0.669 vs. 0.542, estimated difference = 0.127 [0.067, 0.186]), while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth.
| Comments: | Accepted at NeurIPS 2026 (Evaluations and Datasets Track). 50 pages, 9 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2605.25240 [cs.CL] |
| (or arXiv:2605.25240v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.25240 arXiv-issued DOI via DataCite |
Submission history
From: Russell Yang [view email]
[v1]
Sun, 24 May 2026 19:52:39 UTC (3,637 KB)
[v2]
Tue, 2 Jun 2026 02:06:08 UTC (3,776 KB)
[v3]
Thu, 4 Jun 2026 00:43:30 UTC (3,777 KB)
[v4]
Thu, 5 Nov 2026 06:46:06 UTC (3,676 KB)
来源：arXiv cs.CY · arxiv.org