arXiv cs.AI· Rohan Deb, Mohammad Ghavamzadeh, Arindam Banerjee·· 13 小时前AI 评分20
保守上下文对决赌博机：基线约束下的偏好学习
Conservative Dueling Bandits: Preference Learning under Baseline Constraints
AI 导读
研究提出保守上下文对决赌博机设定，要求每轮展示中较优选项的累积得分不低于可信基线的 (1-α) 比例，且约束仅依赖效用差、无需绝对奖励。针对线性效用提出 MaxInP-BC，全程高概率安全，遗憾为 Õ(d√T/κ) 加不随轮数增长的保守代价；针对一般效用类提出 BTIGW-BC，将在线逻辑回归 oracle 的保证转为累积安全证书，回退次数有限且遗憾次线性。
正文 · 原文
Abstract:Contextual dueling bandits learn from pairwise comparisons, the same feedback used to fine-tune language models from human preferences. Standard algorithms explore aggressively and can show users two poor options. We study conservative contextual dueling bandits, which require that, at every round, the cumulative score of the better displayed item stay above a $(1-\alpha)$ fraction of a trusted baseline's. An item's score is its probability of beating the best available item, so the constraint depends only on utility differences and needs no absolute rewards. This also changes how safety can be certified: we show that transferring the safety test of the conservative bandit algorithm CLUCB unchanged can incur linear regret, even when the baseline's utilities are known exactly. For linear utilities, we develop MaxInP-BC, which certifies scores through utility gaps and, when it must fall back, pairs the baseline with an informative partner so that fallback rounds still drive learning. It is safe with high probability at all times, with regret $\tilde{O}(d\sqrt{T}/\kappa)$ plus an additive cost of conservatism that does not grow with the horizon. For general utility classes, we develop BTIGW-BC, which turns the guarantee of an online logistic-regression oracle into cumulative safety certificates. It is safe at all times, makes finitely many fallbacks, and has sublinear regret whenever the oracle's logistic regret is sublinear. In experiments, both satisfy the constraint, which their unconstrained counterparts often violate, at a moderate cost in regret.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.36708 [cs.LG] |
| (or arXiv:2610.36708v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36708 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rohan Deb [view email]
[v1]
Tue, 27 Oct 2026 06:25:04 UTC (317 KB)
来源：arXiv cs.AI · arxiv.org