arXiv cs.AI· Wenkai Li, Wenyi Hong, Zhuoyi Yang, Jie Tang·· 2 天前AI 评分29
视觉生成能否扩展统一多模态模型的知识边界？
Can Visual Generation Expand What Unified Multimodal Models Know?
AI 导读
研究提出 Knowledge Transfer Probing（KTP）协议，通过图像生成训练向统一多模态模型（UMMs）注入新知识，再用纯文本问答测试其可获取性。
正文 · 原文
Abstract:Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what pretrained models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image--Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.21459 [cs.CV] |
| (or arXiv:2610.21459v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21459 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wenkai Li [view email]
[v1]
Thu, 15 Oct 2026 08:03:24 UTC (1,121 KB)
来源：arXiv cs.AI · arxiv.org