arXiv cs.AI· Jiazheng Li, Huaqing Zhang, Jing Xu, Tianle Cai, Jingzhao Zhang·· 12 小时前AI 评分38
LARE：通过损失重加权在固定数据配比下实现快速 LLM 预训练
Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting
AI 导读
研究者提出 LARE（Loss-Adaptive Reweighting under fixed Exposure），在不改变采样数据配比、无需额外搜索权重的情况下，依据运行中的损失统计自动调整各领域权重。
正文 · 原文
Abstract:Large language models often acquire specialized capabilities through successive pretraining stages, but these gains can come at the cost of forgetting earlier knowledge. In this work, we investigate whether such trade-offs can be improved by changing how strongly each domain contributes to learning while keeping its data exposure fixed. We find that, in controlled knowledge--mathematics experiments, loss reweighting produces a single checkpoint that surpasses the best knowledge and reasoning scores attained along either stage trajectory. This reveals an independent control that data-mixture ratios alone do not capture. Building on this finding, we propose LARE (Loss-Adaptive Reweighting under fixed Exposure), which automatically adjusts domain weights from running loss statistics. Motivated by a theoretical analysis of gradient noise, its normalized inverse-square-root rule balances relative loss scales without changing the sampled mixture or requiring additional runs to search for weights. Across knowledge, mathematics, and code, \Method{} improves all five evaluated benchmarks for a 1.5B model trained on 240B tokens, averaging +7.84 pp over a matched direct mixture, including gains of +17.06 pp on GSM8K and +9.00 pp on MATH. These findings demonstrate that adjusting domain loss scales can improve joint capabilities without additional domain exposure.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32141 [cs.LG] |
| (or arXiv:2610.32141v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32141 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiazheng Li [view email]
[v1]
Fri, 23 Oct 2026 04:09:46 UTC (822 KB)
来源：arXiv cs.AI · arxiv.org