arXiv cs.AI· Duc Anh Vu, Thong Nguyen, Hoang Pham, Ponhvoan Srey, Luu Anh Tuan·· 11 小时前AI 评分24
OGC-DPO：面向多目标对齐的目标条件主动偏好学习框架
Objective-Conditioned Active Preference Learning for Multi-Objective Alignment
AI 导读
OGC-DPO 是一个面向 LLM 多目标对齐的主动偏好学习框架，将每个（回复对，目标）组合视为独立查询，用策略自身的隐式梯度衡量信息量，无需外部奖励模型，并优先选择最能扩展各目标梯度覆盖的查询。在不同目标数量的实验中，相同标注成本下 OGC-DPO 优于现有主动偏好学习方法。分析还表明，有信息量的偏好冲突并非一律有害，反而能提供有用的策略训练信号。
正文 · 原文
Abstract:Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives. Active preference learning can reduce this cost by selecting the comparisons whose labels are expected to change the policy most. Existing remedies either lack an objective-aware query selection mechanism for the multi-objective setting or regard conflicting preferences as harmful, therefore excluding them from the training data. We introduce OGC-DPO (Objective-conditioned Gradient Coverage DPO), an active learning framework that treats each (response pair, objective) combination as a separate query and measures its informativeness through the policy's own implicit gradient, without relying on an external reward model. OGC-DPO then selects the queries that most expand each objective's gradient coverage. Empirical experiments across varying numbers of objectives demonstrate the effectiveness of OGC-DPO over existing active preference learning methods under the same annotation cost. Further analysis demonstrates that informative conflicts, rather than being uniformly harmful, provide a useful policy training signal.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.05506 [cs.LG] |
| (or arXiv:2611.05506v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05506 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Duc Anh Vu [view email]
[v1]
Tue, 3 Nov 2026 19:04:16 UTC (213 KB)
来源：arXiv cs.AI · arxiv.org