arXiv cs.AI· Andrew Gao, William Gao·· 1 天前AI 评分48
bf16 舍入静默丢弃大部分 RL 策略更新：一种隐藏的训练-推理不匹配与一行代码的无偏修复
bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix
AI 导读
语言模型 RL 后训练中，bf16 舍入会丢弃小于半个 bf16 间距的更新，Qwen3-1.7B 上每步仅 0.4-0.5% 的权重发生变化，生成数据的策略落后优化器 50-150 步。
正文 · 原文
Abstract:Reinforcement learning (RL) post-training of language models keeps fp32 master weights in the optimizer and serves rollouts from a bf16 copy produced by round-to-nearest conversion. At the learning rates in common use, most per-step updates are smaller than half a bf16 spacing and the conversion discards them: on Qwen3-1.7B only 0.4-0.5% of served weights change per step, and the policy that generates the training data lags the optimizer by 50-150 steps. Existing mismatch corrections cannot see this lag because they compare two copies downstream of the same rounding. We propose Fair Cast: export the master with an unbiased rounding rule (stochastic rounding or error feedback) and compute the training loss on the served copy. The change is one line at the export call, adds no hyperparameters and costs 22-38 ms per export of 1.7B parameters. Across 25 matched seed-pairs on Qwen3-0.6B to 4B with AdamW and Muon, Fair Cast is ahead on 18, tied on 6 and behind on 1, with a mean gain of 6.1 points; on zebra puzzles it reaches 0.57 +/- 0.16 over four runs where round-to-nearest collapses to 0.33 +/- 0.10. It also makes the saved fp32 checkpoint agree with the served model, and it exposes the implicit lag of round-to-nearest as a regularizer that can be restored explicitly when a task needs it.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41073 [cs.LG] |
| (or arXiv:2610.41073v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41073 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Andrew Gao [view email]
[v1]
Thu, 29 Oct 2026 23:40:32 UTC (2,250 KB)
来源：arXiv cs.AI · arxiv.org