arXiv cs.CY· Louise Yorke·· 13 小时前AI 评分44
聚合分数掩盖了什么：系统改变 Arena-Hard 的 LLM 评判设置揭示了什么
Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals
AI 导读
研究用 500 条 Arena-Hard v2.0 提示词、固定两个模型的回答，让 GPT-4.1 在 criteria、输出格式与任务框架的八种组合下担任评判，并在 temperature 0 下重复运行。
正文 · 原文
Abstract:LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions. We show that similar aggregate scores can conceal substantial changes in individual verdicts, both when the same setup is repeated and when the instructions change. Using 500 Arena-Hard v2.0 prompts with two models' answers held fixed, we run GPT-4.1 as the judge under all eight combinations of criteria, output format and task framing, and repeat every setup unchanged at temperature zero on a pinned model version. Rerunning the default setup moves the aggregate score by about one point but changes the win/tie/loss outcome on 21.2% of prompts. Instruction changes introduce further sensitivity: in one setup, which asks for the verdict without reasoning and also drops several other instructions, outcomes change on 28.2 percentage points more prompts than the two setups' unchanged reruns do on average, and the aggregate score falls by 4.09 points. In another, where the criteria prioritise accuracy, outcomes change on 6.7 percentage points more prompts by the same comparison, while the aggregate score barely moves. A stable aggregate score therefore does not establish that individual verdicts are repeatable. This distinction matters when those verdicts are used as preference labels or to assess individual responses. Evaluations that rely on individual verdicts should repeat each setup they use and report both aggregate scores and prompt-level agreement.
| Comments: | Accepted at the JUDGe workshop (Can We Trust the Judge?), NeurIPS 2026 (non-archival). 18 pages, 3 figures. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.30512 [cs.CL] |
| (or arXiv:2610.30512v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30512 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Louise Yorke [view email]
[v1]
Wed, 21 Oct 2026 22:46:14 UTC (191 KB)
来源：arXiv cs.CY · arxiv.org