arXiv cs.CL· Donghao Zeng (Mila, McGill University), Nishant Subramani (Carnegie Mellon University), Tamar Rott Shaham (MIT CSAIL), Benno Krojer (Mila, McGill University, Northeastern University)·· 13 小时前AI 评分31
研究：跨模态引导向量同时存在于图像与文本生成中
Cross-modal Steering Vectors Exist Across Both Image and Text Generation
AI 导读
研究在多个统一多模态模型中发现，引导向量可零样本双向迁移：在文本 token 上找到的向量加到图像 token 上同样能引导生成，反之亦然。即便对某一模态高度独有的概念（如正式与口语化语言），向量也能在另一模态引发一致且可解释的效果，如正式服装与设计。这种迁移还有几何特征：尽管文本与图像表示间存在明显的模态鸿沟，跨模态迁移性强的引导向量在同一概念上倾向指向相似方向。
正文 · 原文
Abstract:Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality. Our study goes beyond simple correlational alignment (e.g. representational similarity) because steering is fundamentally causal. We study cross-modal steering in several unified multimodal models (UMMs), since they possess shared parameters for language and image generation, focusing on these two modalities. Across diverse steered concepts such as emotion and object size, we show that steering vectors transfer zero-shot in both directions (text-to-image & image-to-text). Even for concepts that are highly unique to one modality (e.g. formal vs. colloquial language), steering vectors often induce a consistent, interpretable effect in the other modality, such as formal clothing and design. This transfer has a geometric signature too: despite a clear modality gap separating text and image representations, steering vectors with strong cross-modal transferability tend to point in similar directions for the same concept. Our findings shed light on how vision and language interact inside unified multimodal models and point towards more general, modality-agnostic approaches for controlling their behavior.
| Comments: | Pre-print. Code: this https URL Data: this https URL Demo: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.36269 [cs.CV] |
| (or arXiv:2610.36269v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36269 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Benno Krojer [view email]
[v1]
Mon, 26 Oct 2026 20:42:35 UTC (26,672 KB)
来源：arXiv cs.CL · arxiv.org