arXiv cs.CL· Hare Krishna·· 6 天前AI 评分15
掩码文本重建中的语料采样分歧预测
Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction
AI 导读
研究提出一种基于邻近 token 对计数的平滑预测器，可在不接触目标语料的情况下，从独立参考语料预测两份同源语料训练出的预测器之间的期望分歧。在 TinyStories、WikiText-103 和 CNN/DailyMail 上，语料从 512 增至 2,048 个 64-token 块，分歧仅下降 5–40%，远低于 1/n 缩放暗示的 75%。
正文 · 原文
Abstract:If a second text corpus were collected from the same population, how much would a predictor's outputs change? We study this question for masked-text reconstruction using a smoothed predictor built from nearby token-pair counts and evaluated over the full 50,257-token GPT-2 vocabulary. Treating each source document as one observation, we derive a first-order formula for the expected disagreement between predictors fitted to two independent corpora. The formula accounts for variable document length, random context counts, and correlations across token offsets, and it can be evaluated from a separate reference corpus without access to either target corpus. On TinyStories, WikiText-103, and CNN/DailyMail, growing each corpus from 512 to 2,048 chunks of 64 tokens lowers disagreement by only 5--40\%, far short of the 75\% implied by $1/n$ scaling. The forecast tracks this slow decline, which we trace to smoothing of rarely seen contexts. Forecast accuracy depends less on how these correlations are modeled than on the size of the reference corpus: small references systematically underpredict disagreement. A source-split jackknife that uses only reference data reduces this calibration error by 57--59\%. After this correction, a simpler smoothed-multinomial approximation is most accurate when the reference corpus is smaller than the target corpus, and a nonlinear source bootstrap is most accurate when it is larger. Finally, small masked diffusion language models trained on the same corpora show that the count-based forecasts partly identify which inputs are most sensitive to the corpus, but miss the neural models' much faster decline in disagreement.
| Comments: | 12 pages + Appendices |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.12987 [cs.CL] |
| (or arXiv:2610.12987v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12987 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hare Krishna [view email]
[v1]
Fri, 9 Oct 2026 04:40:47 UTC (154 KB)
来源：arXiv cs.CL · arxiv.org