arXiv cs.CY· Gayane Ghazaryan, Esra D\"onmez·· 12 小时前AI 评分39
更强的奖励模型未必偏见更少：超越标准奖励基准的社会对齐评估
Better Reward Models Are Not Always Less Biased: Social Alignment Beyond Standard Reward Benchmarks
AI 导读
研究将奖励模型评估扩展到偏见、安全、道德与伦理推理四个社会性领域，发现标准基准上的高分可能掩盖社会行为的重要差异。模型在规避部分刻板印象的同时会强化另一些，聚合偏见分数会掩盖这种相反倾向；更强的刻板印象规避还可能以牺牲上下文忠实性为代价。在广泛奖励模型集合上，标准基准表现与社会对齐行为仅呈不一致关联。
正文 · 原文
Abstract:Reward models are central to LLM alignment, yet current benchmarks reveal little about how they represent diverse and sometimes competing social preferences. We extend reward model evaluation to four socially consequential domains: bias, safety, morality, and ethical reasoning. Our analysis shows that strong benchmark performance can obscure important differences in social behavior. Models may avoid some stereotypes while reinforcing others, and aggregate bias scores can hide these opposing tendencies. We also find that stronger stereotype avoidance can come at the expense of contextual faithfulness, revealing a trade-off between social bias mitigation and sensitivity to context. Across a broad set of reward models, standard benchmark performance is only inconsistently related to socially aligned behavior. These results show that stronger reward models are not uniformly better at navigating pluralistic social expectations and motivate more direct evaluation of the social preferences encoded in reward models.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2605.05003 [cs.CL] |
| (or arXiv:2605.05003v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05003 arXiv-issued DOI via DataCite |
Submission history
From: Esra Dönmez [view email]
[v1]
Wed, 6 May 2026 15:04:23 UTC (8,599 KB)
[v2]
Fri, 23 Oct 2026 15:12:30 UTC (6,152 KB)
来源：arXiv cs.CY · arxiv.org