arXiv cs.CL· Nicholas Roberts, Ahmed Ahmed, Suhas Kotha, Percy Liang, Yu Wang, Alex Renda, Nitya Mani·· 12 小时前AI 评分47
用计算与数据约束重写合成数据的 Scaling Law
Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints
AI 导读
研究提出一条重写 Scaling Law，用模型规模、训练 token 数、重写 token 占比和自然数据轮数预测损失，并计入生成重写所耗算力。
正文 · 原文
Abstract:Synthetic rewrites, where a language model rewrites a natural document e.g. into a target style, are now a standard ingredient of pretraining corpora. However, critical aspects of their use remain poorly understood, such as how much compute should be spent rewriting training data and whether a model trained on rewrites can outperform the rewriter. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites. We fit this scaling law in two settings. In the compute-limited setting, natural data is plentiful and rewrites come from a third-party rewriter trained with more compute than our largest run. We find that the estimated optimal rewrite fraction increases with compute, from under $10\%$ at $3\times10^{18}$ FLOPs to $32\%$ at $10^{22}$ FLOPs. Our scaling forecast extrapolates $33\times$ in FLOPs to train a 22B-parameter model, whose final loss is within $1.1\%$ of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute. In the data-limited setting, we assume we only have a fixed seed set of natural data and that our rewriter is a model of the same size as the target, pretrained on that seed set. Again, we find the optimal rewrite fraction increases with the compute budget. We also demonstrate that a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B. Our results show that synthetic rewriting can be studied through the scaling law lens: that optimal methodological choices are predictable and scaling does not necessarily break down beyond the rewriter's capabilities.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.31867 [cs.LG] |
| (or arXiv:2610.31867v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31867 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nicholas Roberts [view email]
[v1]
Thu, 22 Oct 2026 19:56:45 UTC (1,639 KB)
来源：arXiv cs.CL · arxiv.org