arXiv cs.AI· Chengtao Jian, Kai Yang·· 12 小时前AI 评分20
TPO：面向 OOD 偏好对齐的三层分布鲁棒学习框架
TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization
AI 导读
研究者提出 TPO，一个针对 OOD 偏好对齐的三层分布鲁棒框架，用分层矩形模糊集同时建模组级、提示级和偏好级三层分布不确定性。以 Wasserstein 距离实例化后，目标函数由闭式偏好级 margin 惩罚、提示级对抗嵌入扰动和最坏情况组重加权组成，各自由独立半径控制。该工作给出有限样本收敛保证，并在实验中验证 TPO 优于现有鲁棒 DPO 方法，论文已被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Direct Preference Optimization (DPO) minimizes empirical risk over observed preference comparisons, implicitly assuming that the training preference distribution matches the deployment preference distribution. In practice, this i.i.d. assumption often fails, as preference distribution shift exhibits a natural hierarchical structure across three levels: group-level shift, prompt-level shift, and preference-level shift. Such multi-level distributional mismatch poses a critical challenge for safe LLM deployment, yet existing robust DPO methods typically either model uncertainty over the joint preference distribution or focus on individual sources of shift, leaving the hierarchical structure of OOD preference shifts underexplored. We propose TPO, a tri-level distributionally robust framework for OOD preference alignment, which jointly models all three levels of distributional uncertainty through a hierarchically structured rectangular ambiguity set. By instantiating it with Wasserstein distance, we derive a tractable and interpretable objective that combines a closed-form preference-level margin penalty, prompt-level adversarial embedding perturbation, and worst-case group reweighting, each governed by an independent radius. We establish finite-sample convergence guarantees and empirically validate the effectiveness of TPO over existing robust DPO methods.
| Comments: | Accepted at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.28229 [cs.LG] |
| (or arXiv:2610.28229v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28229 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chengtao Jian [view email]
[v1]
Tue, 20 Oct 2026 13:57:52 UTC (95 KB)
来源：arXiv cs.AI · arxiv.org