arXiv cs.AI· Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh, Jack Ji, Jithin Pradeep·· 3 天前AI 评分29
格式即筛选器：评分格式决定 LLM 评判器能表达什么
The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express
AI 导读
研究固定候选池、评判提示词与选择预算，用三种 LLM 评判器以五种评分格式对 15,406 条 UltraFeedback 候选数据打分，微调 66 个模型并用四种评判器评估。0–100 分制实际只用到 7–12 个等距档位，仅为 1–5 分制的两到三倍；二值判定在 77–78% 的提示词中最高分并列，导致多数选择实为随机。基于评判器的数据筛选流程应报告四项指标。
正文 · 原文
Abstract:When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1--5 rating, 0--100 score, or comparison) is rarely justified or ablated. Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges. A 0--100 scale is worth only 7--12 equally used levels, two to three times a 1--5 scale rather than twenty, and a binary verdict leaves the top score tied in 77--78\% of prompts, so most selections are made at random. These properties are set by the format almost independently of the judge. Kendall's $\tau_b$ hides them: it ranks binary best while $\tau_a$ and top-1 agreement rank it last, because it discounts the ties a selector must break at random. Both comparative formats inherit a slot bias whose sign differs across judges. Downstream, curated data beat random selection under every evaluation judge, robust to length and markdown adjustment, but no ordering among formats is detectable. We propose four quantities that judge-based curation pipelines should report.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.17158 [cs.AI] |
| (or arXiv:2610.17158v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17158 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lovedeep Gondara [view email]
[v1]
Mon, 12 Oct 2026 17:36:39 UTC (57 KB)
来源：arXiv cs.AI · arxiv.org