arXiv cs.AI· Licheng Pan, Haocheng Yang, Yuzhou Chen, Yichen Sun, Yuan Lu, Yunfei Teng, Jiajun Fan, Guangyi Liu, Zhixuan Chu, Hao Wang·· 2 天前AI 评分27
CleanRM：用少量干净反馈为 RLHF 噪声反馈做奖励建模
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
AI 导读
CleanRM 通过少量干净反馈，借助最优传输校准逐样本噪声比例，并用噪声校正代理损失训练奖励模型，理论上证明其目标对干净风险无偏。实验显示，CleanRM 优于当前最先进的去噪基线，并能带来更有效的下游 RLHF 对齐。
正文 · 原文
Abstract:Feedback noise is pervasive in preference data, arising from subjective human judgments, annotation mistakes, and imperfect LLM judges. Standard reward modeling trained on such feedback yields a biased estimate of the clean risk. Although classical denoising methods are directly applicable to noisy feedback learning, they generally assume a uniform noise ratio across samples. However, this assumption fails for feedback data, where noise is intrinsically sample-wise. To address this challenge, we propose CleanRM, which uses a few clean feedback to calibrate sample-wise noise ratios via optimal transport. Specifically, CleanRM aligns the joint distribution of preferences reverted from noisy feedback with that of clean feedback and then trains the reward model using a noise-corrected surrogate loss. Theoretically, we prove that the resulting objective is unbiased with respect to the clean risk. Extensive experiments demonstrate that CleanRM outperforms state-of-the-art denoising baselines and yields more effective downstream RLHF alignment.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.21138 [cs.AI] |
| (or arXiv:2610.21138v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21138 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Licheng Pan [view email]
[v1]
Thu, 15 Oct 2026 02:49:51 UTC (343 KB)
来源：arXiv cs.AI · arxiv.org