arXiv cs.AI· Ning Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi·· 6 天前AI 评分26
超越成对比较：语言模型正在隐式优化偏好图——GraphDPO 提出图结构偏好优化
Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
AI 导读
研究者提出 GraphDPO，将 DPO 从成对比较推广到有向无环偏好图，通过图结构的 Plackett-Luce 目标聚合支配邻域监督，并保留 DPO 作为特例。该方法用等价类构造处理并列与不可比回复，缺失边贡献零损失，每提示复杂度保持线性，并可选插入已验证解作为支配节点配合退火调度稳定训练。在推理与程序合成任务上表现优于成对及列表式对齐目标。
正文 · 原文
Abstract:Direct Preference Optimization (DPO) aligns language models using pairwise preference comparisons, offering a simple and effective alternative to Reinforcement Learning (RL) from human feedback. However, in many practical settings, training data consists of multiple rollouts per prompt, inducing rich preference structure that pairwise DPO fails to exploit. Collapsing such data into independent pairs discards transitivity, introduces redundant or conflicting supervision, and can lead to unstable optimization. We propose Graph Direct Preference Optimization (GraphDPO), a structured generalization of DPO that operates over directed acyclic preference graphs induced by rollout rankings. GraphDPO encodes dominance relations as edges and optimizes a graph-structured Plackett--Luce-inspired objective that aggregates supervision over dominated neighborhoods, propagating the transitive structure of the input preferences while recovering standard DPO as a special case. GraphDPO naturally handles tied and incomparable responses through an equivalence-class construction: responses with identical preference signals form graph layers, incomparable responses share a layer without an edge, and absent edges contribute zero loss, preventing the spurious gradients that strict total-order objectives incur. Under this layered construction, GraphDPO maintains linear per-prompt complexity via efficient log-sum-exp aggregation. We further incorporate optional ground-truth anchoring by inserting verified solutions as dominant nodes and applying an annealed schedule that stabilizes early training while gradually relaxing oracle supervision. Experiments on reasoning and program synthesis tasks demonstrate superior performance, suggesting that graph-structured preference modeling is a scalable and robust alternative to pairwise and listwise alignment objectives.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.08037 [cs.LG] |
| (or arXiv:2605.08037v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08037 arXiv-issued DOI via DataCite |
Submission history
From: Ning Liu [view email]
[v1]
Fri, 8 May 2026 17:26:14 UTC (167 KB)
[v2]
Thu, 8 Oct 2026 18:14:25 UTC (183 KB)
来源：arXiv cs.AI · arxiv.org