arXiv cs.AI· Erhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu, Francesco Quinzan, Chengchun Shi·· 2 天前AI 评分26
面向大语言模型的双重稳健对齐算法
Doubly Robust Alignment for Large Language Models
AI 导读
针对 RLHF 中偏好模型、参考策略或奖励函数易被错误设定导致微调效果不佳的问题，研究者提出一种双重稳健偏好优化算法，只要偏好模型或参考策略其中之一被正确设定即可保持一致性，无需两者同时正确。该算法在理论与实践中均表现出优于现有 SOTA 算法的性能和稳健性，代码已公开，论文被 NeurIPS 2025 接收。
正文 · 原文
Abstract:This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at this https URL
| Comments: | Accepted to NeurIPS 2025 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2506.01183 [cs.LG] |
| (or arXiv:2506.01183v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.01183 arXiv-issued DOI via DataCite |
Submission history
From: Erhan Xu [view email]
[v1]
Sun, 1 Jun 2025 21:34:37 UTC (973 KB)
[v2]
Tue, 28 Oct 2025 20:58:26 UTC (1,018 KB)
[v3]
Tue, 27 Oct 2026 08:23:56 UTC (1,003 KB)
来源：arXiv cs.AI · arxiv.org