arXiv cs.CL· Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin, Hao Tang, Hung-yi Lee·· 12 小时前AI 评分31
研究揭示大型音频语言模型中的音文对齐：跨模态概念神经元
Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons
AI 导读
研究提出大型音频语言模型（LALMs）中存在音频与文本表征对齐的共享子空间，并在三个 LALM 和四类声学属性上识别出负责同时表征音频与文本的跨模态神经元。因果干预实验表明，对这些神经元进行引导和停用不仅能验证其对预测的必要性，还能主动操纵模型输出。
正文 · 原文
Abstract:While the language model backbone is pretrained on text alone, large audio-language models (LALMs) can, to a great extent, recognize acoustic attributes, such as emotion, speaker gender, and spoken language. However, it is unclear how the audio representation is integrated, if at all, with the other text representation before the model produces text. We hypothesize that there is a shared subspace where the audio and text representations are aligned. To test this hypothesis, we identify cross-modal neurons that are responsible for representing both audio and text. We find alignment between audio and text neurons to various degrees across three LALMs and four acoustic attribute types. Causal interventions, steering and deactivating the neurons, confirm that the cross-modal neurons are not just necessary for prediction but can also be manipulated.
| Subjects: | Sound (cs.SD); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2610.28189 [cs.SD] |
| (or arXiv:2610.28189v1 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28189 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tzu-Quan Lin [view email]
[v1]
Tue, 20 Oct 2026 13:27:40 UTC (4,603 KB)
来源：arXiv cs.CL · arxiv.org