arXiv cs.CL· Haoyuan Wu, Aoqi Wu, Jiani Hu, Bei Yu·· 3 天前AI 评分31
原生多模态预训练中视觉内容作为可扩展锚点：arXiv 研究揭示视觉信号如何影响扩展行为
More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training
AI 导读
一项 arXiv 研究通过受控视觉内容消融实验发现，在原生多模态预训练中保留视觉内容可改善图文样本的文本预测，且多模态损失收益随计算量近似呈幂律增长。视觉内容还能提升纯文本描述的空间推理能力，在定位、导航和符号规划任务上增益最持续，同时不改变最优模型规模与数据配比，也不损害纯文本损失和聚合语言分数。
正文 · 原文
Abstract:Native multimodal pre-training trains a single model from scratch on text and multimodal data, enabling it to understand both language and visual inputs. However, how visual signals shape scaling behavior and support capabilities beyond visual understanding remains unclear. We investigate this question through controlled visual-content ablations across model sizes and training budgets. Specifically, we replace image embeddings with content-free padding token embeddings. Retaining visual content improves text prediction on image-text examples without materially changing the lowest text-only loss at a given compute budget, the optimal balance of model size and training data, or aggregate language scores. The relative multimodal loss benefit grows approximately as a power law of compute. To test benefits beyond understanding images, we evaluate spatial reasoning from text descriptions alone. Both conditions improve with model and data scaling. During training of the largest model, visual gains are most sustained in localization, navigation, and symbolic planning, which involve maintaining object configurations or composing spatial updates; gains in spatial relations, geometry, and physical commonsense are less consistent. These findings identify visual content as a scalable anchor for multimodal learning, with benefits that extend beyond visual inputs while preserving aggregate language performance.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19443 [cs.CL] |
| (or arXiv:2610.19443v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19443 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haoyuan Wu [view email]
[v1]
Wed, 14 Oct 2026 02:33:34 UTC (450 KB)
来源：arXiv cs.CL · arxiv.org