arXiv cs.CL· Ming-Hao Hsu, Yuxuan Hu, Shujie Liu, Jinyu Li, Yan Lu, Zhizheng Wu·· 2 天前AI 评分26
文本就够了？文本作为语音 LLM 的通用信息瓶颈
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
AI 导读
研究者提出 Convex Gate（C-Gate），将语音每一帧表示为冻结 token 嵌入的凸组合，用 Whisper-large-v3 编码器加部分适配的 Qwen2.5-7B-Instruct 解码器联合训练语音识别、情感识别与口语推理，单一共享解码器即可完成转录与情感识别。适配后解码器在纯文本基准上指令跟随与 MMLU 下降，GSM8K 与 HellaSwag 持平或超过原模型。
正文 · 原文
Abstract:Large language models (LLMs) provide a powerful reasoning backbone for speech understanding, but integrating continuous acoustic signals into a pretrained LLM remains challenging. A central design question is how to tie speech representations to the LLM's text input space while preserving the continuous variation needed for different speech tasks. In this work, we propose Convex Gate (C-Gate), a speech-LLM bridge that represents each frame as a convex combination of frozen token embeddings. This architectural convex-hull constraint fixes the output codebook while allowing continuous mixtures learned from task supervision. With a Whisper-large-v3 encoder, a partially adapted Qwen2.5-7B-Instruct decoder, and joint training on speech recognition, emotion recognition, and spoken reasoning, C-Gate performs accurate transcription and emotion recognition, our representative semantic and acoustic tasks, through one shared decoder. On text-only benchmarks, the adapted decoder loses instruction-following and MMLU accuracy and matches or exceeds the original model on GSM8K and HellaSwag. We further design analyses and interventions on the trained model to examine where the bridge sequence carries task information and how the decoder reads it. These findings show how fixed text embeddings can support both semantic and acoustic readouts through continuous speech-conditioned mixtures.
| Comments: | NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2606.09366 [cs.CL] |
| (or arXiv:2606.09366v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.09366 arXiv-issued DOI via DataCite |
Submission history
From: Ming-Hao Hsu [view email]
[v1]
Mon, 8 Jun 2026 11:38:40 UTC (197 KB)
[v2]
Wed, 14 Oct 2026 16:52:10 UTC (210 KB)
来源：arXiv cs.CL · arxiv.org