跳到正文
arXiv cs.AI· Zhengyuan Fan, Zhonghua Wu, Tianyu Cheng, Qun Chen·· 10 小时前AI 评分31

奖励增益失衡如何影响排序:重新审视基于评分标准的奖励聚合

How Reward Gain Imbalance Shapes Ranking: Rethinking Rubric-Based Reward Aggregation

AI 导读

研究将评分标准间的分数异质性形式化为“奖励增益失衡”,发现即使使用真实标签消除这种失衡,反而会降低排序准确率。作者据此推导出候选奖励差的精确分解,并提出自适应双中心裁剪聚合(DCC),通过为各评分标准的分数差设定统一上界来限制极端分数影响,同时保留连续判断信息。

正文 · 原文

View PDF HTML (experimental)

Abstract:Rubric-based reinforcement learning constructs rewards for aligning large language models with human preferences, typically by equally aggregating continuous scores that a judge assigns to individual criteria. However, scoring heterogeneity across criteria can distort candidate rankings. Existing work has focused on improving judges, learning criterion weights, or refining aggregation, whereas the interaction between scoring heterogeneity and ranking remains insufficiently understood. We formalize this heterogeneity as \emph{reward gain imbalance}: each criterion enters the aggregated reward with its own reward gain, the gap between its expected scores on satisfied and unsatisfied responses. Counterintuitively, our experiments show that removing this imbalance, even with ground-truth labels, lowers ranking accuracy. To explain this, we derive an exact decomposition of the reward difference between any two candidates into a base satisfaction signal, a reward gain imbalance term, and a residual term, showing that reward gain imbalance can attenuate or amplify the base signal, and finding that misrankings usually concentrate on pairs where it attenuates. Guided by this analysis, we propose the adaptive \textbf{D}ual-\textbf{C}enter \textbf{C}lipping aggregation (DCC), which bounds each criterion's contribution to the reward difference instead of simply removing the imbalance. Using only the scores of the current candidate group, it determines a clipping interval that imposes a common upper bound on per-criterion score differences between candidates, limiting the excessive influence of extreme scores while preserving continuous judgment information....
Subjects: Machine Learning (stat.ML); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24005 [stat.ML]
  (or arXiv:2611.24005v1 [stat.ML] for this version)
  https://doi.org/10.48550/arXiv.2611.24005

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zhengyuan Fan [view email]
[v1] Wed, 18 Nov 2026 07:56:58 UTC (71 KB)

来源:arXiv cs.AI · arxiv.org