arXiv cs.CL· Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang·· 12 小时前AI 评分31
双模型互审：面向 LLM 对齐的协作奖励建模框架 CRM
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
AI 导读
研究者提出协作奖励建模框架 CRM，让两个奖励模型互相评审对方的样本选择，并结合课程学习保持学习进度同步，从而在线识别可靠偏好。在 40% 偏好噪声下，CRM 将奖励模型在 RewardBench 上的泛化能力最高提升 9.94 分。该框架还可推广到隐式奖励对齐，为从不完美反馈中学习提供通用方案。
正文 · 原文
Authors:Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang
Abstract:Reward models (RMs) are central to aligning large language models (LLMs) with human values, yet noisy preference feedback can induce reward misgeneralization through spurious correlations and overfitting. We characterize the training dynamics of preference pairs and uncover a distinctive optimization signature of noise: persistently high losses, sharp temporal fluctuations, and irregular gradient updates. Our analyses further show that excluding such pairs improves downstream alignment. In practice, however, reliable preferences are not known a priori, and self-filtering based on a model's own training signals can amplify confirmation bias. To address this challenge, we introduce Collaborative Reward Modeling (CRM), an online framework for identifying reliable preferences. CRM couples cross-model peer review with curriculum learning: two RMs evaluate each other's sample selections, while curriculum keeps their learning progress synchronized. Across extensive experiments, CRM improves RM generalization by up to 9.94 points on RewardBench under 40% preference noise. CRM further generalizes to implicit-reward alignment, offering a robust and broadly applicable approach to learning from imperfect feedback.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2505.10597 [cs.LG] |
| (or arXiv:2505.10597v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.10597 arXiv-issued DOI via DataCite |
Submission history
From: Mingxu Chai [view email]
[v1]
Thu, 15 May 2025 10:58:20 UTC (4,803 KB)
[v2]
Mon, 19 May 2025 03:28:14 UTC (6,702 KB)
[v3]
Fri, 16 Oct 2026 12:42:25 UTC (5,227 KB)
来源：arXiv cs.CL · arxiv.org