arXiv cs.AI· Jia Wei, Ziyi Qiu, Yi Xu, Xuehai Qian·· 11 小时前AI 评分31
FlexFloor：通过弹性专家并行实现资源高效的强化学习后训练
Resource-Efficient Reinforcement Learning Post-Training through Elastic Expert Parallelism
AI 导读
FlexFloor 是一个在同步响应生成阶段提前释放计算资源的系统，通过联合调整响应调度与活跃 worker 组，在保持专家可用性的同时逐步释放 worker。相比 verl 基线，其单位占用计算时间生成的响应 token 数提升约 65.40%，响应生成吞吐提升 52.74%，rollout reward 变化很小。
正文 · 原文
Abstract:Reinforcement learning post-training improves the capabilities of large language models, but generating training responses is a major computational bottleneck. In mixture-of-experts models, workers that finish their assigned responses may remain occupied because their experts are still needed by workers processing longer responses. Consequently, approximately one-third of occupied worker time in the baseline occurs after local response generation finishes. Here we present FlexFloor, a system that enables earlier resource release during synchronous response generation while preserving the expert computation required by ongoing responses. FlexFloor jointly adapts response scheduling and the active group of workers. It uses historical response lengths and memory constraints to guide batching and prompt placement, then progressively releases workers while maintaining expert availability and correct communication. Compared with the verl baseline, FlexFloor generates approximately 65.40% more response tokens per unit of occupied computing time and increases response generation throughput by 52.74%, with little observed variation in rollout reward. By making reinforcement learning post-training more efficient, FlexFloor supports faster development and iterative improvement of large language models and AI agents.
| Comments: | 19 pages including references and appendices, 5 figures |
| Subjects: | Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2611.06984 [cs.AI] |
| (or arXiv:2611.06984v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06984 arXiv-issued DOI via DataCite |
Submission history
From: Jia Wei [view email]
[v1]
Wed, 4 Nov 2026 15:36:14 UTC (1,055 KB)
来源：arXiv cs.AI · arxiv.org