arXiv cs.CL· Giuseppe Curci, Stefano Simonazzi, Andrea Molinari, Andrea Zugarini·· 13 小时前AI 评分31
研究揭示 Delta Learning 假设的局限：DPO 单对生成器偏好数据存在系统性偏差
Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization
AI 导读
针对 DPO 的 delta-learning 假设，研究从理论与实验证明固定模型对会引入与回复质量无关、可预测偏好标签的生成器依赖偏差，导致单对数据训练的 DPO 持续逊于多模型偏好数据，即使大模型回复质量更高，并将差距归因于奖励黑客式的训练动态。增加生成器数量即使无外部评判模型也能部分缓解该效应，生成器多样性是偏好数据构建的关键因素。
正文 · 原文
Abstract:Direct Preference Optimization (DPO) typically relies on preference pairs generated from diverse model pools and ranked by an external LLM judge. The recently proposed delta-learning hypothesis suggests that comparable supervision can instead be obtained from a single pair of models by treating the larger model's response as preferred. We investigate the limits of this approach across model families and scales. We show, both theoretically and empirically, that fixed model pairs introduce systematic, generator-dependent biases that are predictive of the preference label independent of response quality. Consequently, DPO trained on single-pair data consistently underperforms multi-model preference data, despite the larger generator producing higher-quality responses, and we trace this gap to training dynamics consistent with reward hacking on these biases. We further show that simply increasing the number of generators, even without an external judge, is enough to partially mitigate this effect. Our results highlight generator diversity as a critical factor in preference-data construction.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.37058 [cs.LG] |
| (or arXiv:2610.37058v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37058 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Giuseppe Curci [view email]
[v1]
Tue, 27 Oct 2026 11:24:50 UTC (4,289 KB)
来源：arXiv cs.CL · arxiv.org