arXiv cs.CL· Joseph Lee, Yidi Huang, Dokyoon Kim, Shu Yang, Li Shen·· 12 小时前AI 评分38
研究发现辅助视角是 LLM 预训练知识获取的关键
Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
AI 导读
一项被 EMNLP 2026 Findings 接收的研究发现，辅助视角（概念性重述）是 LLM 预训练知识获取的关键因素。在固定 token 预算下，将部分 token 从改写句分配给辅助视角，即使对事实回忆也有提升；辅助视角的效果不取决于生成它的模型强弱，改写仅在较小 batch size 下有帮助。研究还从逐层偏置与压缩角度给出机制解释，并为合成数据增强提供具体方案。
正文 · 原文
Abstract:Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We find that auxiliary views, conceptual reformulations, are key to this process. We design controlled experiments to isolate this. First, holding the token budget fixed, allocating tokens from paraphrases to auxiliary views improves learning even for factual recall, which we use near-verbatim sentences from the original document to test. Second, the effectiveness of auxiliary views is not contingent on the strength of the model that generates them. Third, we establish that paraphrasing is helpful but only at smaller batch sizes. Fourth, we identify types of knowledge that aid learning in the presence of prior knowledge gaps. Finally, we examine how these effects manifest mechanistically via layer-wise biases and compression. Together, our findings suggest that auxiliary representations of knowledge are a key factor in the success of pre-training, and explains why data diversity matters. Further, our results provide clear recipes for synthetic data augmentation along with implications on the self-improvement in LLMs.
| Comments: | Accepted to Findings of EMNLP 2026; Updated with refined writing, new figures, corrected tables and plots |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.04180 [cs.CL] |
| (or arXiv:2609.04180v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.04180 arXiv-issued DOI via DataCite |
Submission history
From: Joseph Lee [view email]
[v1]
Thu, 3 Sep 2026 17:57:02 UTC (708 KB)
[v2]
Tue, 20 Oct 2026 16:21:50 UTC (1,263 KB)
来源：arXiv cs.CL · arxiv.org