arXiv cs.AI· Yijing Ke, Tianle Cai, Ruosong Wang·· 14 小时前AI 评分32
对齐舍入位置可稳定 LLM 强化学习：RoPE、SwiGLU 与 RMSNorm 的 BF16 精度实验
Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning
AI 导读
研究发现，将训练侧在 RoPE、SwiGLU 和 residual RMSNorm 三个区域的舍入位置与推理侧对齐，可防止默认 BF16 基线下的训练崩溃。在四个 Qwen 系列模型上，三区域全部对齐有效，仅对齐 RoPE 路径也足以稳定训练，且无需改动推理引擎、矩阵乘与注意力 kernel 及学习目标。
正文 · 原文
Abstract:Reinforcement learning for large language models often uses separate systems for inference and training. Even with identical weights and BF16 tensors, implementation differences can make an intended on-policy update off-policy and destabilize training. Existing remedies align the entire inference and training stacks, change global precision, or modify the learning algorithm. We ask whether aligning selected parts of the execution stack can stabilize BF16 RL while other implementation differences remain. We align training-side rounding locations with inference in three regions---RoPE, SwiGLU, and residual RMSNorm---by retaining selected intermediates in FP32 until their BF16 outputs. Across four Qwen-family models, aligning all three regions prevents the collapse observed under the default BF16 baseline; RoPE-path alignment alone also suffices in the studied runs, despite remaining numerical discrepancies. The interventions keep the inference engine, matrix-multiplication and attention kernels, and learning objective unchanged.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32951 [cs.LG] |
| (or arXiv:2610.32951v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32951 arXiv-issued DOI via DataCite |
Submission history
From: Yijing Ke [view email]
[v1]
Fri, 23 Oct 2026 17:45:03 UTC (1,307 KB)
来源：arXiv cs.AI · arxiv.org