arXiv cs.CL· Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, Kilian Q. Weinberger·· 12 小时前AI 评分38
数据受限训练的规定性缩放定律
Prescriptive Scaling Laws for Data Constrained Training
AI 导读
针对高质量数据日益稀缺的现状，研究者提出一种含单参数过拟合惩罚项的缩放定律，修正了假设每个训练 token 唯一的 Chinchilla 定律。该定律指出重复训练超过某一点后收益转负，算力应更多投入模型容量，按推荐配置训练可在数据受限场景提升性能。案例显示强权重衰减（λ=1.0）可将过拟合系数降低约 70%，为数据受限下最优权重衰减比常规大一个数量级提供了缩放定律解释。
正文 · 原文
Abstract:Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay ($\lambda=1.0$) reduces this coefficient by approximately 70\%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.
| Comments: | COLM 2026 |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.01640 [cs.LG] |
| (or arXiv:2605.01640v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.01640 arXiv-issued DOI via DataCite |
Submission history
From: Justin Lovelace [view email]
[v1]
Sat, 2 May 2026 23:14:20 UTC (407 KB)
[v2]
Wed, 4 Nov 2026 23:24:06 UTC (430 KB)
来源：arXiv cs.CL · arxiv.org