arXiv cs.CL· Yaxin Luo, Jiacheng Cui, Xiaohan Zhao, Xinyi Shang, Jiacheng Liu, Xinyue Bi, Zhaoyi Li, Zhiqiang Shen·· 2 天前AI 评分27
LLMSurgeon：诊断大语言模型的预训练数据混合
LLMSurgeon: Diagnosing Data Mixture of Large Language Models
AI 导读
研究者提出 LLMSurgeon 框架，将数据混合手术（DMS）形式化为逆问题，仅凭目标 LLM 的生成文本即可在预设分类体系下估计其预训练语料的领域级分布。该方法不直接聚合分类器输出，而是估计校准的软混淆矩阵并求解约束逆问题，以纠正系统性领域混淆、还原潜在混合先验。
正文 · 原文
Abstract:The pretraining data mixture of Large Language Models (LLMs) constitutes their "digital DNA", shaping model behaviors, capabilities, and failure modes. Yet this composition is rarely disclosed, making post-hoc auditing of data combination or provenance difficult. In this work, we formalize Data Mixture Surgery (DMS): given only generated text from a target LLM, estimate the domain-level distribution of its pretraining corpus under a predefined taxonomy. We propose $\textbf{LLMSurgeon}$, a strong framework that casts DMS as an inverse problem under the label-shift assumption. Rather than directly aggregating classifier outputs, LLMSurgeon estimates a calibrated $\textit{soft}$ confusion matrix and solves a constrained inverse problem to correct systematic domain confusion and recover the latent mixture prior. To evaluate, we introduce $\textbf{LLMScan}$, a recipe-verifiable evaluation suite built from open-source LLMs with transparent pretraining mixtures. Across LLMScan, LLMSurgeon recovers domain mixtures with high fidelity under fixed protocols. Our work presents a practical, post-hoc approach for auditing the digital DNA of foundation models without access to their training data.
| Comments: | ACL 2026 Main. Code at this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.30348 [cs.CL] |
| (or arXiv:2605.30348v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.30348 arXiv-issued DOI via DataCite |
Submission history
From: Zhiqiang Shen [view email]
[v1]
Thu, 28 May 2026 17:59:53 UTC (2,289 KB)
[v2]
Thu, 15 Oct 2026 12:11:31 UTC (2,289 KB)
来源：arXiv cs.CL · arxiv.org