arXiv cs.CL· Wenqian Cui, Xiao-Hui Li, Daxin Tan, Qiyong Zheng, Irwin King·· 11 小时前AI 评分31
TextPro-SLM：从输入侧缩小模态差距，让语音 LLM 具备韵律感知的文本 LLM 能力
Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
AI 导读
TextPro-SLM 从输入侧缩小语音大模型的模态差距，在 3B 和 7B 规模下均取得领先 SLM 中最低的模态差距，并在副语言理解任务上表现强劲。该模型结合统一语音编码器 WhisperPro（同步输出文本 token 与韵律嵌入）与保留原 TLM 语义能力的 LLM 主干，仅用约 1,000 小时 LLM 训练音频即达成上述效果。论文已被 NeurIPS 2026 接收，代码已开源。
正文 · 原文
Abstract:Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient. Code is available at this https URL
| Comments: | Accepted by NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2605.05927 [cs.CL] |
| (or arXiv:2605.05927v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05927 arXiv-issued DOI via DataCite |
Submission history
From: Wenqian Cui Mr. [view email]
[v1]
Thu, 7 May 2026 09:32:05 UTC (290 KB)
[v2]
Fri, 8 May 2026 01:27:50 UTC (290 KB)
[v3]
Fri, 30 Oct 2026 11:02:03 UTC (290 KB)
来源：arXiv cs.CL · arxiv.org