arXiv cs.CL· Leif Sieben, Yoel Zimmermann·· 3 天前AI 评分34
预训练真的让化学语言模型学会化学了吗？
Does Pretraining Teach Chemical Language Models Chemistry?
AI 导读
研究系统测试了一系列化学语言模型（CLM）在 64 个分子性质预测数据集上的表现，发现预训练带来的下游收益并不随规模增长，性能在梯度提升基线之下就饱和。只有基于传统理化描述符的监督预训练能稳定提升下游性能，部分预训练方式甚至让模型不如随机初始化编码器。模型虽能解析分子结构，但表现并不优于随机初始化模型，说明预训练至多学到类描述符性质与低层 token 统计，超出经典分子特征的化学知识出奇地少。
正文 · 原文
Abstract:Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Chemical Physics (physics.chem-ph) |
| Cite as: | arXiv:2610.18651 [cs.LG] |
| (or arXiv:2610.18651v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18651 arXiv-issued DOI via DataCite |
Submission history
From: Yoel Zimmermann [view email]
[v1]
Tue, 13 Oct 2026 16:14:35 UTC (555 KB)
[v2]
Wed, 14 Oct 2026 17:58:29 UTC (555 KB)
来源：arXiv cs.CL · arxiv.org