arXiv cs.CL· Louis Est\`eve, Christophe Servan, Thomas Lavergne, Agata Savary·· 12 小时前AI 评分31
法国版 ModernBERT 预训练研究：多样性采样让数据量减少 94%、训练时间缩短 73%
A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT
AI 导读
一项针对法语 ModernBERT 的研究发现，随机采样与多样性驱动采样都能在保持性能的前提下将预训练数据集缩减最多 94%、预训练时间缩短最多 73%。在部分任务上，多样性采样模型经 head-only 微调估计的内在质量比随机采样高出最多 10 分。该研究在固定算力预算下对比了多种多样性采样算法，并用最优算法预训练了多个法语 ModernBERT 模型，在多项法语基准上完成微调与评测。
正文 · 原文
Abstract:Diversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons to investigate the impact of diversity on pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining at least comparable performance. We compare diversity-driven sampling algorithms, and we use the best one to pre-train several ModernBERT models on French with a fixed compute budget. We fine-tune and evaluate them on a variety of French benchmarks. We compare them with models pre-trained on randomly sampled data of commensurate size, with the same compute budget. We find that both random and diversity-driven sampling may reduce the pre-training dataset by up to 94% and the pre-training time by up to 73% while maintaining performance. Moreover, in some tasks, the inherent quality of models, estimated via head-only fine-tuning, is up to 10 points higher with diversity sampling than with random sampling.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2602.22014 [cs.CL] |
| (or arXiv:2602.22014v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2602.22014 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: ACL 2026, pages 34168-34181, San Diego, California, United States. Association for Computational Linguistics |
| Related DOI: | https://doi.org/10.18653/v1/2026.findings-acl.1707
DOI(s) linking to related resources |
Submission history
From: Louis Estève [view email]
[v1]
Wed, 25 Feb 2026 15:29:30 UTC (360 KB)
[v2]
Fri, 23 Oct 2026 09:13:36 UTC (376 KB)
来源：arXiv cs.CL · arxiv.org