arXiv cs.CL· Seonghyeon Cho, Chanjun Park·· 23 小时前AI 评分31
通过表征子空间理解 Omni 模型中的跨模态不一致性
Understanding Cross-Modal Inconsistency through Representation Subspaces in Omni Models
AI 导读
研究从答案位置表征中估计低维 Answer Subspace,在三个 Omni 模型和两个多模态推理任务上分离答案相关成分与其余表征。预测跨模态不一致的信息主要位于 Answer Subspace 之外,集中在一个可复现的二维表征中,且无法用固定模态偏移解释。仅对单一模态扰动 Answer Subspace 即可诱发跨模态分歧,对两个模态施加相同扰动则基本保持一致,说明可线性解码不等于行为有效。
正文 · 原文
Abstract:Omni Models process audio, vision, and text within a unified language backbone, yet often produce different answers for semantically equivalent inputs across modalities. Existing representation-level analyses do not distinguish between information that predicts whether disagreement will occur and the representation whose perturbation affects the model's answer. In this paper, we estimate a low-dimensional Answer Subspace from answer-position representations and use it to separate answer-related components from the remaining representation. Across three Omni Models and two multimodal reasoning tasks, information predictive of cross-modal inconsistency lies mainly outside the Answer Subspace, concentrating in a reproducible two-dimensional representation that is not explained by a fixed modality offset. Matched interventions reveal distinct behavioral roles: perturbing the Answer Subspace substantially changes the model's predictions, whereas perturbing the disagreement representation, like a random subspace of the same dimensionality, leaves them almost unchanged. Our key finding is that cross-modal disagreement can be induced by misaligning modality-specific answer representations: perturbing the Answer Subspace for only one modality induces disagreement, whereas applying the same perturbation to both largely preserves agreement. Thus, the representation that predicts disagreement is distinct from the answer-level representation through which disagreement can be induced, showing that linear decodability does not imply behavioral efficacy.
| Comments: | Accepted for EACL 2027 Main |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.22619 [cs.CL] |
| (or arXiv:2611.22619v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22619 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Seonghyeon Cho [view email]
[v1]
Tue, 17 Nov 2026 11:35:06 UTC (1,496 KB)
来源:arXiv cs.CL · arxiv.org