arXiv cs.CL· Xiangchi Yuan, Xingjian Diao, Zheyuan Liu, Chunhui Zhang, Dachuan Shi, Kejing Xia, Zhenbang Du, Shenglong Yao, Soroush Vosoughi, Wenke Lee·· 13 小时前AI 评分31
PAPO：面向多奖励策略优化的优先级保持优势聚合方法
Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization
AI 导读
针对 RL 中加性奖励聚合可能出现的“优势优先级反转”问题，研究者提出优先级感知的优势聚合方法 PAPO，在主辅优势一致时保留辅助监督、冲突时平滑衰减辅助信号，融合中保持主目标符号。该方法适用于 GRPO 的序列级优势和 actor-critic PPO 的 token 级优势，在多个领域和模型上均取得最佳主目标表现，同时辅助目标保持最佳或具竞争力，代码已开源。
正文 · 原文
Abstract:LLM post-training often optimizes multiple objectives with unequal priorities, such as correctness and reasoning efficiency. However, the widely used additive reward aggregation in RL can violate these priorities: when primary and auxiliary advantages conflict, a sufficiently large auxiliary signal can overturn the preference induced by the primary objective. We term this problem \emph{advantage priority inversion}. Although reward-level gating can protect priority more directly, it still removes potentially useful auxiliary distinctions among rollout samples that fail the primary condition. To address this trade-off, we propose Primary Advantage Policy Optimization (\textbf{PAPO}), a priority-aware advantage aggregation method that asymmetrically combines a primary and an auxiliary advantage. PAPO retains auxiliary supervision when the objectives agree and smoothly attenuates conflicting auxiliary signals, preserving the primary sign during fusion for \emph{primary-consistent ordering}. This rule applies to sequence-level advantages in GRPO and token-level advantages in actor--critic PPO. Experiments on multiple domains and models show that PAPO consistently achieves the best primary objective while maintaining best or competitive auxiliary-objective performance across both GRPO- and PPO-based optimization. \href{this https URL}{Code is available}.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34862 [cs.LG] |
| (or arXiv:2610.34862v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34862 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiangchi Yuan [view email]
[v1]
Mon, 26 Oct 2026 05:26:04 UTC (461 KB)
来源：arXiv cs.CL · arxiv.org