arXiv cs.AI· Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu, Haoliang Li·· 14 小时前AI 评分34
面向噪声偏好数据的大语言模型无偏对齐框架
Unbiased Alignment for Large Language Models with Noisy Preferences
AI 导读
研究提出无偏对齐理论框架，包含 Unbiased Reward Model（URM）损失与 Unbiased Direct Preference Optimization（UDPO）损失，通过数学修正偏好噪声带来的失真，使模型可直接从噪声数据集中无偏训练，无需干净的 ground-truth 监督。
正文 · 原文
Abstract:The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: this https URL.
| Comments: | Accepted by ICML 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.03248 [cs.LG] |
| (or arXiv:2607.03248v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.03248 arXiv-issued DOI via DataCite |
Submission history
From: Jialiang Wang [view email]
[v1]
Fri, 3 Jul 2026 12:04:43 UTC (1,699 KB)
[v2]
Mon, 26 Oct 2026 15:40:42 UTC (1,700 KB)
来源：arXiv cs.AI · arxiv.org