arXiv cs.CL· Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida, Kim Sung-Bin, Toshimitsu Uesaka, Takashi Shibuya, Kyeongyoon Lee, Tae-Hyun Oh, Yuki Mitsufuji·· 13 小时前AI 评分31
ST-AudioLM：面向动态声源的时空音频语言建模
Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
AI 导读
研究者提出 ST-AudioQA 时空音频问答数据集与基准，基于一阶 Ambisonic（FOA）渲染的静态与移动声源构建，每个场景包含声源身份、活动、方向、距离与运动元数据。
正文 · 原文
Abstract:Sound events have semantic identities, locations, and trajectories, yet existing audio-language models typically encode clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To bridge this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and QA about what is sounding, where it is, how it moves, and how sources relate. We further introduce ST-Audio Encoder, which jointly learns event semantics and time-resolved source trajectories, and ST-AudioLM, which exposes these audio tokens to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the balance between semantic preservation and localization and yields stronger reasoning than static spatial and localization-oriented baselines. Beyond the controlled benchmark, ST-AudioLM shows zero-shot transfer to real recordings, with further gains after light adaptation. Project page: this https URL.
| Comments: | EMNLP 2026 Main Conference |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.14141 [cs.SD] |
| (or arXiv:2606.14141v2 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2606.14141 arXiv-issued DOI via DataCite |
Submission history
From: Hyun-Bin Oh [view email]
[v1]
Fri, 12 Jun 2026 05:58:31 UTC (273 KB)
[v2]
Tue, 27 Oct 2026 10:34:08 UTC (1,477 KB)
来源：arXiv cs.CL · arxiv.org