arXiv cs.AI· Yuhang Han, Dingning Liu, Yujie Chen, Yaojie Zhang, Jingwei Song, Zequn Yang, Xuanming Shang, Li Ding, Zhijiang Guo, Linfeng Zhang·· 12 小时前AI 评分29
OVC：面向多模态强化学习的成果条件化视觉信用分配框架
From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning
AI 导读
研究者提出 Outcome-Conditioned Visual Credit（OVC），一个可插拔框架，将视觉关联度量与 token 级信用分配分离，适用于统一分配响应级优势的策略优化方法。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but uniformly assigning a single response-level advantage to all tokens overlooks the distinct roles of visual observations and subsequent reasoning. Visual affinity alone is insufficient: predictive states may align with both image and prompt-context representations, and upweighting valid visual observations can amplify their negative-advantage updates when later reasoning fails. To address this issue, we propose \textbf{Outcome-Conditioned Visual Credit (OVC)}, a plug-in framework that separates visual-association measurement from token-level credit assignment and is adaptable to policy-optimization methods with uniform response-level advantage allocation. Relative Visual Encoding constructs a context-relative visual coordinate by contrasting each predictive state's affinity to image and prompt-context representations. Outcome-conditioned Evidence Calibration combines this coordinate with uncertainty-scaled predictive features and calibrates their contributions to outcome-discriminative evidence through mean contrasts between positive- and negative-advantage responses, scaled by regularized within-side variation. Visual association thus informs credit through observed outcome differences rather than directly determining update priority. Normalized Credit Allocation conditions the calibrated evidence on the response's advantage sign to produce positive token weights with unit global mean. These detached weights preserve advantage signs while modulating individual token-gradient contributions. OVC reuses existing rollouts without auxiliary reward models, process annotations, or inference-time modules. Instantiated through GSPO-token, OVC retains sequence-level clipping and consistently improves six-benchmark mean accuracy over GSPO across three models, with gains of 1.86-2.39 percentage points.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32324 [cs.AI] |
| (or arXiv:2610.32324v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32324 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yujie Chen [view email]
[v1]
Fri, 23 Oct 2026 08:22:16 UTC (527 KB)
来源：arXiv cs.AI · arxiv.org