arXiv cs.AI· Zhengyang Luo, Xiaoyu Qiu, Zhiheng Tan, Yuchen Liang, Yingbin Liang, Liyue Shen·· 3 天前AI 评分22
统一 VLM 内部机制解析：从注意力探测到 HeadTuned-CFG 引导调优
Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation
AI 导读
研究通过探测注意力头发现，统一 VLM 在 T2I 与 I2T 生成中均存在对条件变化敏感与不敏感的两类注意力头，该功能分化模式跨样本稳定。基于此提出免训练推理干预方法 HeadTuned-CFG，在注意力头级别选择并调优条件引导，在两种生成方向上均取得一致收益，T2I 生成较强基线提升显著。
正文 · 原文
Abstract:Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how the backbone processes the condition in either direction remains poorly understood. We study this question from the perspective of conditional guidance: we probe every attention head to show the activation difference between two branches of classifier-free guidance (CFG) with different conditional guidance. On two unified VLMs of autoregressive and masked-diffusion paradigms, the probing reveals a subset of heads that are more sensitive to the change of condition while others are agnostic to it. This characteristic pattern holds in both T2I and I2T generation and is largely stable across data samples, demonstrating the internal mechanism of unified VLMs. Motivated by this, we propose a new inference-time intervention approach for unified multimodal generation, named \textit{HeadTuned-CFG}, which uses the probed condition-change awareness to select and tune the conditional guidance at the attention-head level. Experiments show consistent benefits in both multimodal generation directions of unified VLMs without additional training, and in particular substantial improvements over strong baselines in T2I generation. These results further support the key finding of our probing analysis: the observed pattern of functional specialization is an intrinsic property of unified VLMs.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15227 [cs.AI] |
| (or arXiv:2610.15227v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15227 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhengyang Luo [view email]
[v1]
Sat, 10 Oct 2026 23:40:27 UTC (1,432 KB)
来源：arXiv cs.AI · arxiv.org