arXiv cs.CL· Yueying Li, Jingxuan Fan, Kiant\'e Brantley, Sham M. Kakade, Hanlin Zhang·· 6 天前AI 评分37
无监督多模态奖励建模：冻结 Qwen3-VL 中间层实现无标注偏好训练
Unsupervised Multimodal Reward Modeling
AI 导读
研究提出一种无需人类偏好标注的多模态奖励模型：在冻结的 Qwen3-VL 主干中间层上训练线性头，让网络标题得分高于同一标题的扰动版本，标注数据仅用于选层。该中间层选择比最后一层高 5.8 分，32B 规模下在三个基准上追平或超过 GPT-4o，VL-RewardBench 领先 6.6 分，仅需数千参数和约千分之一微调算力。
正文 · 原文
Abstract:Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer. The layer is the decisive choice, worth 5.8 points over the last layer, and one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks at 32B, 6.6 points ahead on VL-RewardBench, from a few thousand parameters at about a thousandth of fine-tuning compute. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. Because the backbone is frozen, the objective is convex, and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6.4 points in hallucination judging. Retraining the head on the policy's own verifier-labeled candidates cuts caption hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark.
Together, an intermediate-layer probe, free same-image perturbations, and a verifier-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.12707 [cs.CL] |
| (or arXiv:2610.12707v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12707 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yueying Li [view email]
[v1]
Thu, 8 Oct 2026 20:38:25 UTC (957 KB)
来源：arXiv cs.CL · arxiv.org