arXiv cs.CL· Jeonghoon Kim, Woojin Chung, Woomin Song, Cheonbok Park, Nako Sung, Jinwoo Shin·· 13 小时前AI 评分34
预训练数据的词汇印记：UCE 可无模型预测零样本基准表现
The Lexical Imprint of Pre-training Data
AI 导读
研究提出词级一元交叉熵（UCE）这一与分词器无关、无需模型的指标，用于衡量预训练语料与基准的词汇对齐程度。在覆盖 10 个零样本基准、4 个预训练语料和 3 个模型规模的受控从零预训练中，UCE 越低（对齐越高）的语料在各基准上零样本准确率越高。文档级筛选训练文档可带来机制预测的准确率提升，提示词级按领域对齐路由草稿模型则相对通用草稿基线取得可测的吞吐提升。
正文 · 原文
Abstract:How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a reference corpus and a target benchmark. Likelihood training shapes the model's predictive distribution from the pre-training corpus, and zero-shot scores under likelihood-based evaluation depend in part on the lexical alignment between that corpus and the benchmark target. Cleanly tracing how the lexical imprint of pre-training data scales with corpus--benchmark alignment requires controlled from-scratch pre-training that varies corpus identity while holding the training recipe fixed. Across this controlled sweep, spanning $10$ zero-shot benchmarks, $4$ pre-training corpora, and $3$ model scales, we find that corpus--benchmark lexical alignment, UCE, consistently tracks zero-shot accuracy on every benchmark, with corpora more aligned with the benchmark (lower UCE) achieving higher accuracy. The same signal extends from measurement to control at finer scopes. At the document level, strengthening the imprint by selecting training documents to maximize lexical alignment with the benchmark produces the accuracy gains predicted by the mechanism. At the prompt level, matching the imprint per prompt by routing each prompt to the drafter whose domain corpus is most lexically aligned with it produces measurable throughput gains over a generalist-drafter baseline. Taken together, pre-training corpora leave a measurable lexical imprint on base-model zero-shot behavior, and UCE provides a model-free probe of lexical alignment between reference and target data.
| Comments: | Neurips2026 Camera-ready version |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2602.10657 [cs.CL] |
| (or arXiv:2602.10657v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2602.10657 arXiv-issued DOI via DataCite |
Submission history
From: Jeonghoon Kim [view email]
[v1]
Wed, 11 Feb 2026 09:01:25 UTC (94 KB)
[v2]
Mon, 26 Oct 2026 06:39:59 UTC (1,576 KB)
来源：arXiv cs.CL · arxiv.org