arXiv cs.CL· Yiran Zhang, Hanbo Huang, Jielan Li, Zhen Zhao, Shiyu Liang, Hongxia Hao·· 2 天前AI 评分22
PACE：用成对 token 对比与自适应轨迹置信度增强多模态 RLVR
Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence
AI 导读
针对多模态 RLVR 中组相对策略优化把序列级优势广播到每个 token、且自蒸馏信号无法区分图像依据与语言捷径的问题，研究者提出 PACE，通过四种教师视角下的成对 token 对比判断 token 支持是否依赖图像，并用自适应轨迹置信度调节信用重分配幅度。PACE 在三个骨干模型上取得最佳平均准确率，并在 Qwen3-VL-8B-Instruct 上于所有评测的多模态推理基准排名第一。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but group-relative policy optimization broadcasts a single sequence-level advantage to every token of a response, although errors usually localize to a few decisive steps. Recent self-distilled variants refine credit with a teacher--student likelihood gap. However, this signal cannot tell whether it relies on the image or on a language shortcut, and is applied uniformly even when the teacher is uninformative. To address this, we propose Pairwise token contrast and Adaptive trajectory ConfidencE (PACE). Pairwise token contrast scores each response under four teacher views, crossing a correct or incorrect answer hint with the image present or removed, and compares the two hints for every token, which cancels shifts shared by both hints and reveals whether the token's support relies on the image. Adaptive trajectory confidence measures how well the teacher predicts each response given the correct hint and the image, normalizes this score within the batch, and uses it to set how far the token contrast redistributes credit within that response. PACE achieves the best average accuracy on three backbones and ranks first on all evaluated multimodal reasoning benchmarks with Qwen3-VL-8B-Instruct. Experiments show that both components matter: replacing pairwise token contrast with a single-view signal underperforms our method, while adaptive trajectory confidence adds further improvement.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.21671 [cs.CL] |
| (or arXiv:2610.21671v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21671 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yiran Zhang [view email]
[v1]
Thu, 15 Oct 2026 10:55:47 UTC (993 KB)
来源：arXiv cs.CL · arxiv.org