arXiv cs.AI· Yu Li, Yufa Zhou, Tian Lan·· 1 天前AI 评分34
研究揭示 LLM 强化学习中训练-推理不匹配的起源与演化，并提出 DLC 修正方法
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
AI 导读
研究将 LLM 强化学习中的训练-推理不匹配拆解为推理引擎的数值差异与异步采样的策略滞后两类来源，前者近似恒定、后者经训练器更新形成反馈回路。
正文 · 原文
Abstract:On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the \textbf{training-inference mismatch} between the two makes the training data off-policy. The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other. \textit{We ask which property of a mismatch source decides the correction it needs.} Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into a \textit{numerical discrepancy} of the engine and a \textit{policy lag} of the trainer, and we follow both through training. The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount, and centering the score at each prefix removes its leading bias. The policy lag is produced by the trainer's own updates: the bias it leaves enters the next update, which enlarges the next lag and closes a feedback loop. Importance weighting cuts this loop only while the lag ratio stays concentrated, and on Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half. From this account we derive the \textbf{discrepancy-lag correction} (DLC), which centers the numerical discrepancy, clips the lag ratio against the sampler's weights and resynchronizes the sampler when this effective sample size falls below one half. Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and its resynchronization prevents collapses that the per-token corrections leave and that a KL measured on the sampler's prefixes does not detect.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.40945 [cs.LG] |
| (or arXiv:2610.40945v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40945 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yu Li [view email]
[v1]
Thu, 29 Oct 2026 20:23:04 UTC (581 KB)
来源：arXiv cs.AI · arxiv.org