arXiv cs.AI· Xuhui Zhan, Tyler Derr·· 9 天前AI 评分24
Inverse-LLaVA：用文本到视觉映射重新思考多模态对齐
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
AI 导读
Inverse-LLaVA 反转了传统多模态对齐方向，在解码器注意力内将语言状态投影到视觉特征维度，由模态专属映射生成残差 query、key、value 更新，冻结骨干网络、无需独立对齐阶段。最终 7B 模型在 VQAv2 得 78.45%（LLaVA-LoRA 为 79.13%），VizWiz 得 50.96%（对比 48.56%），TextVQA 为 56.96%（对比 58.47%）。
正文 · 原文
Abstract:Connecting pretrained vision and language models usually involves projecting image features into the language model's input space. Inverse-LLaVA reverses this mapping within decoder attention: language states are projected to the visual feature dimension, and modality-specific maps produce residual query, key, and value updates. Fusion and low-rank adaptation (LoRA) learn jointly from 665K visual instructions, with frozen backbones and no separate alignment stage. Across nine primary benchmark evaluations, the final 7B model approaches two-stage LLaVA-1.5 on several tasks. It scores 78.45% on VQAv2 versus 79.13% for official LLaVA-LoRA, and 50.96% versus 48.56% on VizWiz; TextVQA is lower at 56.96% versus 58.47%. Controlled studies examine fusion components, insertion depth, visual features, and language-model size. Representation analysis shows that the text maps preserve much of the pairwise similarity ordering while changing its geometric spread. Additional paired supervision improves celebrity recognition, while instruction replay repairs caption-induced answer-format failures. Analytical cost expressions and fixed-work profiles separate the additional fusion computation from the omitted alignment stage. These findings establish text-to-vision attention fusion as a practical alternative for instruction-only multimodal adaptation.
| Comments: | 50 pages, 20 figures, including appendices. Substantially revised with retrained models, updated benchmark results, and expanded ablation, efficiency, and representation analyses. Code, model weights, and reproducibility artifacts: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2508.12466 [cs.CV] |
| (or arXiv:2508.12466v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2508.12466 arXiv-issued DOI via DataCite |
Submission history
From: Xuhui Zhan [view email]
[v1]
Sun, 17 Aug 2025 18:36:04 UTC (1,202 KB)
[v2]
Wed, 15 Jul 2026 14:44:35 UTC (1,271 KB)
[v3]
Thu, 8 Oct 2026 15:19:41 UTC (5,021 KB)
来源：arXiv cs.AI · arxiv.org