跳到正文
arXiv cs.AI· Xingzi Xu, Mehmet Saygin Seyfioglu, Karim Bouyarmane·· 4 小时前AI 评分31

大语言模型对齐的 margin 动力学:DPO、IPO 与 SimPO 的理论分析

Margin Dynamics for Large Language Model Alignment

AI 导读

研究者为 LLM 对齐方法提供理论框架:每种成对方法在概率单纯形上定义向量场,投影到单个偏好/非偏好对后化为有闭式解的 margin ODE。闭式解显示 DPO 的 margin 以 O(log t) 无界增长,IPO 收敛到有限值 1/(2τ),标签噪声使 DPO 出现从无界到有界的相变而仅定量影响 IPO。

正文 · 原文

View PDF HTML (experimental)

Abstract:Practitioners choose among LLM alignment methods largely by trial and error. We provide a theoretical foundation: each pairwise method defines a vector field on the probability simplex that governs how the model redistributes mass between preferred and dispreferred completions. Projecting onto a single preferred/dispreferred pair reduces this to a one-dimensional margin ODE with a closed-form solution. These solutions reveal that DPO margins grow without bound (logarithmically as $O(\log t)$ in the gradient-flow limit) while IPO margins converge to a finite $1/(2\tau)$; that label noise induces a qualitative phase transition in DPO (unbounded $\to$ bounded) but affects IPO only quantitatively; and that gradient concentration under asymmetric weighting governs trainability at scale. We confirm all three predictions across five model families spanning four benchmarks. The closed-form solutions also enable principled method design: we prove that adding any label smoothing $\varepsilon > 0$ to SimPO creates a finite margin attractor at $\Delta^* = \beta^{-1}\log((1-\varepsilon)/\varepsilon)$, while $\varepsilon = 0$ (standard SimPO) has no attractor. We test this across three models (8B-9B) and confirm the prediction: the theory-derived configuration (TPO-S, $\varepsilon = 0.05$) is competitive with the strongest baselines, attaining the highest mean IFEval score among the compared methods on all three, validating that the predicted finite attractor is a usable operating point rather than proposing a new state-of-the-art loss.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.21766 [cs.LG]
  (or arXiv:2611.21766v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.21766

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xingzi Xu PhD [view email]
[v1] Mon, 16 Nov 2026 19:02:17 UTC (96 KB)

来源:arXiv cs.AI · arxiv.org