arXiv cs.AI· Riccardo Formenti, Aditya Agrawal, Vikas Garg·· 3 天前AI 评分27
你的偏好模型可信吗？用 Stackelberg 博弈做对齐
Can You Trust Your Preference Models? Stackelberg Games for Alignment
AI 导读
研究将偏好模型集成体之间的分歧作为预测误差的代理，把寻找评估器可能反转偏好的比较样本建模为 Leader 与 Follower 之间的零和 Stackelberg 博弈。
正文 · 原文
Abstract:Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are unavailable during policy training, we use disagreement among an ensemble of preference models as a proxy for prediction error. We formulate this search as a zero-sum Stackelberg game between two opposing policies, a Leader which tries to minimize disagreement, and a Follower which maximizes it. For a general class of finite contextual Stackelberg games, we establish equilibrium existence and, under regularization, uniqueness and a closed-form characterization. We introduce Stackelberg Mirror Descent-Ascent (SMIDA), a two-timescale algorithm with linear last-iterate convergence in the tabular setting, and develop a neural policy-gradient analogue. Experiments with a synthetic oracle support disagreement as an informative proxy for prediction error. On held out data the trained policies outperform reference policies and increase the fraction of comparisons that exceed a disagreement threshold and reverse the oracle's preference.
| Subjects: | Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT) |
| Cite as: | arXiv:2610.15008 [cs.AI] |
| (or arXiv:2610.15008v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15008 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Riccardo Formenti [view email]
[v1]
Sat, 10 Oct 2026 15:41:36 UTC (369 KB)
来源：arXiv cs.AI · arxiv.org