arXiv cs.AI· Ye Lu, Shen Wang, Runze Liu, Zhitao Ou, Haonan Huang, Yihan Yan, Fanghui Sun, Zhaoyang Zhang·· 2 天前AI 评分42
从语音 Token 到被盗声音：攻击者指定文本的说话人模仿攻击
From Speech Tokens to Stolen Voices: Attacker-Chosen-Text Speaker Impersonation via Representation Inversion
AI 导读
研究者提出 STIA（Speech Token Inversion Attack），仅凭一段泄露的语音表征即可用受害者声音生成攻击者指定文本的语音，无需原始音频或转录文本。在 Moshi 表征泄露 3 秒语音的条件下，该方法平均 CAM++ 说话人余弦相似度达 0.78，目标文本词错误率为 3.80%。结果表明语音前端表征会泄露可复用的说话人身份信息，设计此类表征方法时需考虑隐私风险。
正文 · 原文
Abstract:Speech interaction systems have gradually evolved from traditional cascaded systems that use text as intermediate information to end-to-end speech language models that directly process speech representations. In some applications, the speech frontend runs on the user's device and sends speech representations to a remote server for inference, while the original audio remains local. However, these representations may still contain speaker identity information. If an attacker obtains them, it may use this information to imitate the user's voice, threatening their voice privacy. To investigate this risk, we propose \stia, a Speech Token Inversion Attack that generates speech with attacker-specified text in a victim's voice using a single short leaked representation, without access to the original audio or its transcript. Specifically, \stia converts discrete, continuous, or mixed speech representations into conditioning sequences and trains a conditional flow-matching speech generation model using pairs of different utterances from the same speaker in public speech datasets. The model combines speaker characteristics contained in the leaked representation with the specified text to generate speech in the victim's voice. Experiments show that, with a Moshi representation leak corresponding to 3 seconds of speech, \stia achieves an average CAM++ speaker cosine similarity of 0.78 and a target-text word error rate of 3.80\%. These results show that speech frontend representations reveal reusable speaker identity information and highlight the need to consider privacy risks when designing speech frontend representation methods.
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM) |
| Cite as: | arXiv:2610.34688 [cs.SD] |
| (or arXiv:2610.34688v1 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34688 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ye Lu [view email]
[v1]
Mon, 26 Oct 2026 01:24:54 UTC (218 KB)
来源：arXiv cs.AI · arxiv.org