arXiv cs.AI· Fady Rezk, Yuangang Pan, Chuan-Sheng Foo, Xun Xu, Nancy Chen, Henry Gouk, Timothy Hospedales·· 12 小时前AI 评分34
个性化对齐中的奖励模型选择危机：RM 准确率无法预测部署表现
The Reward Model Selection Crisis in Personalized Alignment
AI 导读
研究发现奖励模型（RM）的偏好排序准确率无法预测个性化对齐的部署效果，上游 RM 准确率与下游策略准确率仅呈弱相关（Kendall's tau = 0.08–0.31）。
正文 · 原文
Abstract:Personalized alignment from preference data has focused primarily on improving personal reward model (RM) accuracy, with the implicit assumption that better preference ranking translates to better personalized behavior. However, in deployment, computational constraints necessitate inference-time adaptation such as reward-guided decoding (RGD) rather than per-user policy fine-tuning. This creates a critical but overlooked requirement: reward models must not only rank preferences accurately but also effectively guide generation. We demonstrate that standard RM accuracy fails catastrophically as a selection criterion for deployment-ready personalized rewards. We introduce policy accuracy; a metric quantifying whether RGD-adapted LLMs correctly discriminate between preferred and dispreferred responses and show that upstream RM accuracy correlates only weakly with downstream policy accuracy (Kendall's tau = 0.08--0.31). More critically, we introduce Pref-LaMP the first personalized alignment benchmark with ground-truth user completions, enabling direct behavioural evaluation. On Pref-LaMP, we expose a complete decoupling between discriminative ranking and generation metrics: methods with 20-point RM accuracy differences produce almost identical output quality, and methods with high ranking accuracy can fail to generate behaviorally aligned responses. These findings reveal that the field has been optimizing for proxy metrics that do not predict deployment performance, and that current personalized alignment methods fail to operationalize preferences into behavioral adaptation under realistic deployment constraints. In contrast, we find simple in-context learning (ICL) to be highly effective - dominating all reward-guided methods for models $\geq$3B parameters, achieving $\sim$3 point ROUGE-1 gains over the best reward method at 7B scale.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2512.23067 [cs.AI] |
| (or arXiv:2512.23067v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2512.23067 arXiv-issued DOI via DataCite |
Submission history
From: Fady Rezk [view email]
[v1]
Sun, 28 Dec 2025 20:27:15 UTC (310 KB)
[v2]
Wed, 7 Jan 2026 20:10:03 UTC (301 KB)
[v3]
Tue, 20 Oct 2026 11:28:28 UTC (326 KB)
来源：arXiv cs.AI · arxiv.org