arXiv cs.CL· Dongseong Hwang, Prasanth Yadla, Kaan Elgin, Shifas Padinjaru Veettil, Sivanand Achanta, Dipjyoti Paul, Ramya Rasipuram, Tyler Johnson, Emad Soroush, Chung-Cheng Chiu, Zhifeng Chen·· 12 小时前AI 评分40
苹果 AFM 3 Core Advanced 如何用解耦时序深度扩散 Transformer 实现内存高效音频合成
Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers
AI 导读
苹果为 Siri Expressive Voices 提出一种内存高效的音频合成架构，将语义音频 token 转为 RVQ 表示，并用单个可复用的 DiT 式深度解码器自回归生成全部 RVQ 层级。
正文 · 原文
Authors:Dongseong Hwang, Prasanth Yadla, Kaan Elgin, Shifas Padinjaru Veettil, Sivanand Achanta, Dipjyoti Paul, Ramya Rasipuram, Tyler Johnson, Emad Soroush, Chung-Cheng Chiu, Zhifeng Chen
Abstract:Siri Expressive Voices synthesize rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced, Apple's most powerful on-device foundation model. This work presents the memory-efficient audio synthesis architecture behind that capability: a detokenizer that converts the semantic audio tokens emitted by the foundation model into high-fidelity audio within the tight compute and memory budget of the Scalable Matrix Extension (SME). We convert semantic audio tokens to a residual vector quantization (RVQ) representation with a three-component design, a streaming encoder, a temporal decoder, and a depth decoder, that systematically decouples temporal and depth processing. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing the dedicated per-level decoders of prior multi-decoder architectures, while causal sliding window attention with fixed-window key-value caching yields constant memory complexity independent of sequence length. Deployed on the AMX, the detokenizer sustains roughly 10 ms per generation step, about 16x faster than real time, with a peak runtime memory of only 21 MB and 329 MB of on-device assets, enabling continuous streaming synthesis of 20-320 seconds of audio. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Ablation studies validate the key architectural components, and audio quality assessment confirms that the architecture maintains synthesis fidelity while achieving efficiency gains over existing methods. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.
| Comments: | 11 pages, ICASSP |
| Subjects: | Sound (cs.SD); Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2607.23811 [cs.SD] |
| (or arXiv:2607.23811v3 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2607.23811 arXiv-issued DOI via DataCite |
Submission history
From: Prasanth Yadla [view email]
[v1]
Sun, 26 Jul 2026 19:20:01 UTC (980 KB)
[v2]
Mon, 17 Aug 2026 17:56:26 UTC (981 KB)
[v3]
Mon, 2 Nov 2026 18:54:20 UTC (981 KB)
来源：arXiv cs.CL · arxiv.org