arXiv cs.CL· Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng Cao·· 11 小时前AI 评分31
Li²SAO：用组相对 actor-critic 自举提升轻量单轨迹异步 RL 优化
Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
AI 导读
Li²SAO 通过组相对 actor-critic 自举，解决了轻量版 SAO（LiSAO）因 critic 未训练好而在训练早期落后 GRPO 的问题。该方法在 critic 冷启动阶段用组相对优势训练 actor、用组中心回报训练 critic，之后切换为单轨迹 GAE。实验显示 Li²SAO 在训练早期追平或超越 GRPO，并进一步提升后期性能上限，代码已开源。
正文 · 原文
Authors:Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng Cao
Abstract:Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps. To address this issue, we propose Li$^{2}$SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li$^{2}$SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.40726 [cs.CL] |
| (or arXiv:2610.40726v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40726 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zichen Bai [view email]
[v1]
Thu, 29 Oct 2026 09:02:00 UTC (968 KB)
来源：arXiv cs.CL · arxiv.org