arXiv cs.CL· Zixuan Xu, Haoxuan Chen, Jiangxin Sun, Jian-Fang Hu·· 13 小时前AI 评分20
GPDPO：通过组帕累托支配策略优化从奖励冲突中学习
GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization
AI 导读
研究者提出多奖励策略优化方法 GPDPO，从被冲突过滤的 rollout 中挖掘采样回复间的帕累托支配关系，转化为辅助偏好信号并与原策略目标联合优化，无需额外采样或奖励评估即可复用现有 rollout。在 API-Bank 上跨多个模型规模和奖励配置的实验显示，GPDPO 持续优于现有多奖励优化基线，并降低了组内相对与成对奖励冲突。代码将开源。
正文 · 原文
Abstract:Multi-reward reinforcement learning offers a promising framework for aligning large language models (LLMs) with diverse objectives, but conflicting reward signals can induce ambiguous update directions and hinder stable optimization. Existing approaches address such conflicts through reward reweighting, gradient coordination, or rollout filtering, yet they primarily operate on per-response or group-level optimization signals and underexploit relational information among sampled responses. As a result, conflict-filtered rollouts may still contain useful supervision that is discarded during training. We propose Group Pareto-Dominance Policy Optimization (GPDPO), a multi-reward policy optimization method that recovers pairwise supervision from conflict-filtered rollouts. GPDPO identifies Pareto-dominance relations among sampled responses and converts them into auxiliary preference signals, which are jointly optimized with the original policy objective. This enables more effective reuse of existing rollouts without additional sampling or reward evaluation. Experiments on API-Bank across multiple model scales and reward configurations show that GPDPO consistently outperforms competitive multi-reward optimization baselines. Further analysis demonstrates that GPDPO reduces both group-relative and pairwise reward conflicts, supporting the effectiveness of exploiting relational supervision for multi-reward alignment. Code will be released.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35480 [cs.LG] |
| (or arXiv:2610.35480v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35480 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haoxuan Chen [view email]
[v1]
Mon, 26 Oct 2026 14:11:28 UTC (321 KB)
来源：arXiv cs.CL · arxiv.org