数据策展如何影响 LLM 训练：是提升上限还是加速收敛？

View PDF HTML (experimental)

Abstract:While data curation is widely credited with improving large language model training, a basic question remains unresolved: does curation expand what a model can ultimately learn, or merely reach the same solution faster? In this paper, we formalize this distinction between advancement (improving the attainable performance) and acceleration (reaching the same performance with less data). Through an analytic characterization and real-world LLM experiments, we show that advancement requires a correctable train-target gap. When the baseline training and target objectives coincide, no curation policy improves the endpoint, although curation may still accelerate training. This reframes advancement as a property of the curation task rather than the curation method. Under a common gap-centric view of data mixture and selection, we operationalize this condition as the $\Delta$-test, an observable discrepancy between the uncurated baseline's training and target losses that is computable prior to expensive curation. Across extensive experiments, an appreciably nonzero $\Delta$ predicts persistent gains, whereas gains near $\Delta=0$ tend to diminish as the training budget grows. Beyond individual methods, our findings recenter data curation on the task itself and point to target-set construction as a promising direction for shaping what curation can achieve.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.36983 [cs.LG]
	(or arXiv:2610.36983v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.36983 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jiaxing Wang [view email]
[v1] Tue, 27 Oct 2026 10:34:29 UTC (227 KB)