arXiv cs.AI· Zhen Li, Shuai Zhang, Yanggan Gu, Yiming Zhang, Yang Yu, Mingfa Feng, Congkai Xie, Shuang Yu, Junjie Lai, Hongxia Yang·· 10 天前AI 评分24
TRIAGE：面向原生 NVFP4 强化学习的方向感知失配稳定方法
TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning
AI 导读
TRIAGE 是一种方向感知的稳定方法，通过段级诊断选择性再平衡策略梯度更新，并对残余严重失配做有界修复，同时保留采样器与学习器上的原生 NVFP4 W4A4 前向执行。在 Qwen3-4B 和 Qwen3-30B-A3B 上，该方法在整个训练周期内保持优化稳定，在五个数学推理基准上达到全精度水平，rollout 吞吐量较 BF16 最高提升 2.3 倍。
正文 · 原文
Abstract:Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. In this paper, we characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify. In native NVFP4 runs, we observe an early imbalance between the two amplifying regions, favoring negative-advantage, negative-gap updates. Their tail tokens become concentrated in a small fraction of response segments before mismatch spreads globally. Motivated by these findings, we introduce TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively rebalance policy-gradient updates and applies bounded repair to residual severe mismatch. TRIAGE modifies the optimization objective while retaining native NVFP4 weight-and activation 4-bit (W4A4) forward execution on both the sampler and learner. Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and achieve full precision level performance across five mathematical reasoning benchmarks, while native NVFP4 with TRIAGE provides up to 2.3x higher rollout throughput than BF16.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07043 [cs.LG] |
| (or arXiv:2610.07043v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07043 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhen Li [view email]
[v1]
Mon, 5 Oct 2026 00:53:11 UTC (1,669 KB)
来源：arXiv cs.AI · arxiv.org