跳到正文
arXiv cs.AI· Yihan Wang, N. Asokan·· 10 小时前AI 评分52

研究提出填充式抽取方法,揭示扩散语言模型训练数据泄露被低估

Extracting Training Data from Diffusion Language Models via Infilling

AI 导读

Yihan Wang 与 N. Asokan 提出填充式抽取(infilling extraction)协议,用任意二值掩码刻画扩散语言模型的双向记忆,指出仅做前缀探测会显著低估训练数据抽取风险。

正文 · 原文

View PDF HTML (experimental)

Abstract:Memorization in large language models has been studied almost exclusively through prefix-conditioned extraction, a natural choice for autoregressive models. However, diffusion language models (DLMs) can denoise masked tokens at arbitrary positions. Thus, prefix-only probing reveals only one facet of memorization in DLMs and significantly underestimates the risk of training-data extraction. In order to realistically model extractability of training data in DLMs, we introduce infilling extraction, a data-extraction protocol parameterized by an arbitrary binary mask that subsumes prefix-only probing and accounts for the bidirectional inductive bias of DLMs. Instantiating it on LLaDA-8B and Dream-7B across five extraction modes, three training pipelines, and three corpora covering verbatim and partial leakage, we find that mask geometry governs extractability: edge-conditioned masks extract up to three times more verbatim sequences than prefix-conditioned ones, and bidirectional access opens channels inaccessible in autoregressive models. In particular, an adversary who obtains a PII-redacted copy of the training data can use targeted infilling to raise PII recall on fine-tuned DLMs by 2.5-4.5x over prefix-conditioned extraction, substantially narrowing the gap to a scale-matched autoregressive model and, for LLaDA on email addresses, exceeding it under greedy prefix-conditioned extraction. Tunable parameters for decoding measurably affect extraction performance, while a follow-up supervised fine-tuning stage does not eliminate the prior memorization.
Comments: NeurIPS 2026
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
Cite as: arXiv:2605.24173 [cs.CL]
  (or arXiv:2605.24173v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2605.24173

arXiv-issued DOI via DataCite

Submission history

From: Yihan Wang [view email]
[v1] Fri, 22 May 2026 19:46:08 UTC (361 KB)
[v2] Wed, 18 Nov 2026 16:50:39 UTC (364 KB)

来源:arXiv cs.AI · arxiv.org