CoRP：将奖励扰动整合为单次更新的 LLM 后训练方法

View PDF HTML (experimental)

Abstract:Post-training of language models is commonly framed as a sample-score-update loop implemented by gradient descent. A recent line of work, exemplified by RandOpt, relocates this loop to weight space, sampling Gaussian perturbations around a pretrained model and ensembling the top-K rewarded specialists at inference. While competitive with PPO and GRPO under matched training compute, this prediction-level ensemble incurs K forward passes per test example and does not extend cleanly to free-form generation. We ask whether the rewarded population can instead be folded into a single deployable model, replacing the inference-time ensemble with one consolidated update. A split-half analysis over 25 model-task pairs reveals reproducible low-rank structure in every case. We turn this geometry into CoRP (Consolidating Rewarded Perturbations), a gradient-free operator that combines reward-weighted aggregation, compatibility-aware reweighting, and a held-out validation gate, with no gradient flowing through the language model. Across five language models from 0.5B to 8B and five tasks covering math, code, and creative writing, CoRP improves the base model by 8.1 points on average. Using one tenth of RandOpt's perturbation budget, CoRP exceeds single-inference RandOpt by 6.5 points and recovers more than half of the gain of the 50-pass majority-vote ensemble, at one forward pass per test example.

Comments:	NeurIPS 2026
Subjects:	Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:	arXiv:2605.31494 [cs.CL]
	(or arXiv:2605.31494v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2605.31494 arXiv-issued DOI via DataCite

Submission history

From: Zheyu Zhang [view email]
[v1] Fri, 29 May 2026 16:16:13 UTC (182 KB)
[v2] Fri, 6 Nov 2026 01:45:14 UTC (183 KB)