arXiv cs.CL· Jixuan Leng, Si Si, Fei Wang, Cho-Jui Hsieh, Jiong Zhang, Hsiang-Fu Yu, Inderjit S. Dhillon·· 11 小时前AI 评分22
CONCUR：面向多任务多奖励强化学习的一致性辅助奖励方法
CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning
AI 导读
CONCUR 是一种新的优势估计器，用于多任务、多奖励的 LLM 强化学习训练，通过将各门控辅助奖励的标准化分数替换为满足两个条件的最接近分数，避免辅助奖励反转主奖励的排序。
正文 · 原文
Abstract:When a Large Language Model (LLM) is trained with Reinforcement Learning (RL) on multiple tasks, a task's primary reward can gate auxiliary rewards such as a length reward: on a failing response, an auxiliary cannot exceed the primary. However, group-relative advantage estimators treat a gated auxiliary like any other component, so it can reverse the primary's order, and at the primary's ceiling only auxiliaries order responses. Keeping gated auxiliaries hurts some tasks, while dropping them hurts others. We propose CONCUR, an advantage estimator that replaces the standardized scores of each gated auxiliary with the closest scores that satisfy two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary's ceiling receive equal scores. CONCUR-T applies both conditions throughout training, and CONCUR-A lifts the second for an auxiliary that agrees with the primary's order more often than it reverses it among passing responses. On three models from two families and across math, code, instruction following, and writing, both variants score higher on average than all baselines and than when all the gated auxiliaries are dropped.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.36437 [cs.CL] |
| (or arXiv:2610.36437v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36437 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jixuan Leng [view email]
[v1]
Tue, 27 Oct 2026 00:36:59 UTC (2,374 KB)
来源：arXiv cs.CL · arxiv.org