arXiv cs.AI· Jiahe Feng·· 3 天前AI 评分22
VC-OPD：面向视觉语言推理的视觉反事实在线策略蒸馏
VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
AI 导读
VC-OPD 是一种面向视觉语言推理的在线策略蒸馏方法，通过对比原始图像与破坏实例证据的反事实图像下教师的输出差异，估计 token 级视觉依赖并据此对损失加权，再以保质量归一化稳定更新尺度。该方法在教师监督热启动后执行在线策略蒸馏，在多模态推理基准上提升了视觉推理表现，并产生更有效的 token 级蒸馏。
正文 · 原文
Abstract:GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the original--counterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation.
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; I.2.6; I.4.8 |
| Cite as: | arXiv:2610.14939 [cs.AI] |
| (or arXiv:2610.14939v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14939 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiahe Feng [view email]
[v1]
Sat, 10 Oct 2026 14:10:43 UTC (4,768 KB)
来源：arXiv cs.AI · arxiv.org