arXiv cs.CL· Haodong Wen, Kairong Luo, Tingkai Yan, Wenguang Chen, Kaifeng Lyu·· 13 小时前AI 评分31
数据调度与学习率调度应协同设计：LLM 中期训练中数据混合策略的实证研究
Data Schedules and Learning Rate Schedules Should Be Designed Together
AI 导读
在固定总训练算力与 MT token 预算下，LLM 中期训练是否应完全切换到 MT 数据取决于学习率调度：LR 持续衰减时，提前引入 MT 数据并与 PT 数据混合可降低最终 MT loss，而恒定 LR 下结论反转。消融实验表明，应在 MT 数据进入训练之后再衰减 LR，并提出了解释该依赖关系的理论模型。
正文 · 原文
Abstract:Mid-training (MT) is a training phase in LLM training pipelines that continues pretraining (PT) on smaller datasets of higher value. Taking the PT and MT phases as a whole, training usually starts with only PT data and then transitions completely to MT data. In this work, we view this complete transition from PT to MT data as a special case of a broader class of data mixing schedules that specify how the proportions of PT and MT tokens in each batch change over training. Under fixed total training compute and MT token budgets, we ask: Is this complete transition to MT data optimal for minimizing final MT loss? We show that the answer depends on the learning rate (LR) schedule: one data schedule may outperform another under one LR schedule, but changing only the LR schedule can reverse the outcome. Specifically, when the LR decays throughout continued training, introducing MT data earlier and mixing it with PT data lowers the final MT loss compared with placing all MT data at the end of training, but this result is reversed if we use constant LR. We therefore study how to jointly choose when to introduce MT data and when to decay the LR through ablation studies, and find that it is better to decay the LR only after MT data enters training. Finally, we develop a theoretical model that explains why the optimal data schedule for a fixed MT token budget depends on the LR schedule.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.29348 [cs.LG] |
| (or arXiv:2610.29348v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29348 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kairong Luo [view email]
[v1]
Wed, 21 Oct 2026 07:42:20 UTC (3,204 KB)
来源：arXiv cs.CL · arxiv.org