arXiv cs.CL· Wonsup Shin, Jingu Kim·· 12 小时前AI 评分26
重复数据下随机 Token 替换的收益与代价：已知熵分解与十个真实语料的证据
Benefit and Cost of Random-Token Replacement under Repeated Data: A Known-Entropy Decomposition and Evidence from Ten Real Corpora
AI 导读
研究提出一种将验证损失分解为近似、估计与记忆三项的框架，并在熵率已知的受控数据源上证明随机 Token 替换会把预测拉向语料平均值。在替换有效的场景中，它可将记忆导致的损失上升削减约 60%；在韩日网页文本上，到固定预算结束时消除约三分之一的损失上升，但在最低点处效果小得多。十个真实语料中替换在九个上降低最终损失，仅损害冗余度最高的 census JSON 语料。
正文 · 原文
Abstract:When unique pretraining data is scarce, models see the same tokens many times and are thought to memorize. Random-token replacement, an input-corruption augmentation, is one remedy reported to lower validation loss, but a lower loss alone does not reveal which source of error it reduces. On a controlled source whose entropy rate is known in closed form, we show that replacement shrinks predictions toward the corpus average. We also introduce a decomposition that separates validation loss into approximation, estimation, and memorization terms. This shrinkage view predicts that replacement's bias cost needs no memorization but its benefit does. Memorization raises validation loss after its minimum, so we report the effect at two points, the minimum validation loss and the end of a fixed budget. Where replacement helps, it reduces this rise by about 60%. At the end of the budget, it turns from harmful to helpful as the controlled source becomes less predictable; at the minimum, it helps at no predictability level under the fixed recipe. On Korean and Japanese web text, replacement removes about a third of the loss rise after the minimum by the end of the budget, but its effect at the minimum is much smaller. Across ten real corpora, replacement lowers the final loss on nine. It harms only the census corpus, program-generated JSON that is the most redundant of the ten and much more redundant than any of 64 further prose and code corpora we measure.
| Comments: | 16 pages, 6 figures. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.27590 [cs.CL] |
| (or arXiv:2610.27590v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27590 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wonsup Shin [view email]
[v1]
Tue, 20 Oct 2026 03:57:15 UTC (651 KB)
来源：arXiv cs.CL · arxiv.org