arXiv cs.AI· Aaron Maiwald, Roderick McNeill, Jaeyoung Lee, Matthew A. Clarke, Lennart Justen, Alejandro Acelas, Ivo Andrews, Giles Harper-Donnelly, Adam Moore, Sophie Rose, Oliver M. Crook, Felix Jackson·· 10 小时前AI 评分50
研究:蛋白质语言模型预训练数据过滤的有效性与持久性
The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models
AI 导读
研究以 ESM-2 系列蛋白质语言模型为对象,考察从预训练数据中过滤病毒蛋白的效果。结果显示,过滤会明显削弱模型在病毒蛋白上的掩码语言建模、突变效应预测和长程接触预测表现,而非病毒蛋白性能基本保持;影响集中在感染人类的病毒蛋白上,并随模型从 8M 扩展到 3B 参数而增强,模型越大严格过滤越有效。但用被移除病毒家族的序列做微调即可恢复损失的性能,即使最大模型也只需约 80 美元的 H100 算力。
正文 · 原文
Authors:Aaron Maiwald, Roderick McNeill, Jaeyoung Lee, Matthew A. Clarke, Lennart Justen, Alejandro Acelas, Ivo Andrews, Giles Harper-Donnelly, Adam Moore, Sophie Rose, Oliver M. Crook, Felix Jackson
Abstract:Pretraining filtering is one of the most common safeguards for open-weight language models. Here we study its effectiveness, durability and scalability using the ESM-2 family of protein language models. We find that filtering viral proteins from the pretraining data meaningfully degrades masked language modelling, mutation effect prediction and long-range contact prediction on viral proteins, while largely preserving performance on non-viral proteins. These effects are concentrated among proteins from human-infecting viruses, and grow as we scale models from 8M to 3B parameters. As models increase in size, we find that stringent filtering becomes more effective. However, fine-tuning on sequences from the removed viral families restores the lost performance. Although the cost to recover performance increases with model scale, even for the largest models it costs about 80 USD of H100 time. This work establishes pretraining filtering as moderately effective at selectively degrading performance on viral proteins, but not durable against fine-tuning with modest computational resources and access to relevant data.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.26651 [cs.LG] |
| (or arXiv:2610.26651v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26651 arXiv-issued DOI via DataCite |
Submission history
From: Jaeyoung Lee [view email]
[v1]
Mon, 19 Oct 2026 16:08:59 UTC (962 KB)
[v2]
Wed, 18 Nov 2026 04:18:12 UTC (1,224 KB)
来源:arXiv cs.AI · arxiv.org