自改进训练与思考：将推理与强化学习提前到预训练阶段

Authors:Ellen Xiaoqing Tan, Jack Lanchantin, Zichun Yu, Polina Kirichenko, Shehzaad Dhuliawala, Danwei Li, Thao Nguyen, Jing Xu, Ping Yu, Ilia Kulikov, Sainbayar Sukhbaatar, Jason Weston, Xian Li, Olga Golovneva

View PDF HTML (experimental)

Abstract:Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite data and to judge policy model rollouts, thus introducing thinking and reinforcement earlier in training. We further improve reasoning at all stages by introducing the concept of self-improving post-thinking: improving the model's reasoning and self-judging ability by reasoning after generation. In our experiments, we show these approaches can give strong gains across multiple axes.

Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:	arXiv:2601.21343 [cs.CL]
	(or arXiv:2601.21343v4 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2601.21343 arXiv-issued DOI via DataCite

Submission history

From: Jason Weston [view email]
[v1] Thu, 29 Jan 2026 07:09:30 UTC (4,141 KB)
[v2] Wed, 4 Feb 2026 17:31:10 UTC (4,141 KB)
[v3] Sun, 5 Apr 2026 15:57:55 UTC (4,520 KB)
[v4] Thu, 5 Nov 2026 14:52:55 UTC (8,945 KB)