arXiv cs.AI· Hao Yu·· 11 小时前AI 评分38
序数信用分配：仅凭比较进行多步强化学习
Ordinal Credit Assignment: Multi-Step Reinforcement Learning from Comparisons Alone
AI 导读
研究提出序数信用分配方法，在多步任务中仅用轨迹间两两比较、无需奖励函数或奖励模型即可完成强化学习。其期望更新等价于将策略梯度中的 reward-to-go 替换为对同时步仍在运行群体的胜率，确定性环境下按真实价值排序动作且最优策略稳定。
正文 · 原文
Abstract:Policy-gradient methods need a scalar reward, yet for many tasks no formula captures what we want, while judging which of two outcomes is better is easy. We study reinforcement learning that receives only such comparisons - no reward function, no reward model - in multi-step tasks. Groups of trajectories start from a shared state, and each action is credited by how its reward-to-go compares with those of other trajectories at the same time step. We show the expected update is policy gradient with reward-to-go replaced by its win rate against the population still running at that time, weighted by the chance an opponent is still running; in deterministic environments the update ranks actions by true value and the optimal policy is stable, whereas under stochastic outcomes it prefers actions that win more often rather than those with higher mean. Label noise near ties or from position bias leaves the expected update unchanged; only noise that also flips clear comparisons halves the signal. A critic trained from the same comparisons removes the state confound without changing the expected update. In a pre-registered study over six environments with ten seeds per claim, comparisons cost no detectable performance relative to scalar returns at the same granularity, learning a reward model from the same comparisons is not better, and ten percent of comparisons suffice when spent on a few time points with all pairs. With real LLM judges, comparisons are 7-10 points more accurate than pointwise scores, and the gap grows with the stronger judge. Trained end-to-end with an LLM judge on a requirement the hand-written reward omits, policies satisfy it in all five seeds but land far less often than with oracle comparisons, because the judge is near chance on the near-tied failures dominating early training. Comparisons do not close the gap to PPO with scalar rewards on HalfCheetah.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Statistics Theory (math.ST) |
| Cite as: | arXiv:2605.06375 [cs.LG] |
| (or arXiv:2605.06375v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06375 arXiv-issued DOI via DataCite |
Submission history
From: Hao Yu [view email]
[v1]
Thu, 7 May 2026 14:56:11 UTC (842 KB)
[v2]
Wed, 4 Nov 2026 12:28:07 UTC (92 KB)
来源：arXiv cs.AI · arxiv.org