arXiv cs.CL· Mateusz Modrzejewski, Tomasz Radzikowski·· 13 小时前AI 评分31
成对评估遗漏了什么：奖励分数在不同提示词间的差异
What Pairwise Evaluation Misses: Reward Scores across Prompts
AI 导读
成对评估只比较同一提示词下的两个回答，提示词共享的分数成分会被抵消而无法检验。研究提出基于提示词间方差占比、同源评分回归和提示词替换的人类参照审计方法，应用于18个已发布评分器和6组匹配训练条件；在 HelpSteer2 上，6个奖励模型的提示词间方差占比为 0.34 至 0.64，而人类评分接近零。
正文 · 原文
Abstract:Reward models score responses for preference optimization, filtering and ranking, and pooling scores across prompts needs them to be comparable. Yet pairwise evaluation compares two responses to the same prompt, so any score component shared by a prompt's responses cancels and goes untested. We study this component by comparing between-prompt variation in model scores with the same variation in human ratings of the same responses. We propose a human-referenced audit, built from a between-prompt variance share, a sibling-rating regression and prompt substitutions, and apply it to eighteen released scorers and six matched training conditions. We find that scores cluster by prompt more than the ratings do: on HelpSteer2, six reward models have between-prompt shares of 0.34 to 0.64 against a human share near zero. Replacing the original prompt with a fixed instruction keeps substantial clustering but weakens agreement with prompt-level human ratings, so clustering alone does not show agreement with humans. Part of that variation is useful: subtracting prompt means lowers the helpfulness of globally selected responses, and scores stay sensitive to a wrong prompt even when training shuffles prompt means. We propose that reward evaluation check between-prompt variation, sensitivity to a wrong prompt and usefulness for selection separately.
| Comments: | 31 pages, 5 figures, 33 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.31090 [cs.CL] |
| (or arXiv:2610.31090v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31090 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mateusz Modrzejewski [view email]
[v1]
Thu, 22 Oct 2026 11:00:10 UTC (291 KB)
来源：arXiv cs.CL · arxiv.org