arXiv cs.CL· Maciej Besta, Leonard Schmidt, Pierre Pang, Patrik Okanovic, Robert Gerstenberger, Torsten Hoefler·· 11 小时前AI 评分35
RPEG：可扩展且低成本地训练推理语言模型
RPEG: Training Reasoning Language Models Scalably and Cost-Effectively
AI 导读
研究者提出 RPEG（RLM Pipeline Execution Graph），一种以数据为中心的图中间表示，联合优化推理语言模型 RL 训练流水线的结构与执行，并配套覆盖计算、通信、内存及机架/节点/GPU 层级效应的性能模型。
正文 · 原文
Abstract:Reinforcement learning (RL) is now central to training Reasoning Language Models (RLMs), but RLM pipelines are difficult to scale and costly to run because each RL step couples multiple large models, heterogeneous stages, cross-step state, and complex asynchronous dependencies. Existing systems typically optimize only parts of this design space. We address this complexity by jointly optimizing the pipeline structure as well as its execution using a novel data-centric graph intermediate representation called RLM Pipeline Execution Graph (RPEG). RPEG exposes structural choices such as bounded asynchrony and actor-critic sharing together with execution optimizations such as stage fusion, fine-grained scheduling, and multi-dimensional model parallelism. It is coupled with a comprehensive analytical performance model for RLM training, jointly capturing compute, communication, memory, and hierarchical rack-, node-, and GPU-level effects. Across systems with up to 256 GH200 GPUs, our distributed runtime engine implementing RPEG improves performance by up to 2x over ReaL and reduces step latency by up to 2.48x over verl, while increasing hardware utilization by up to 30%. Overall, RPEG provides a graph-driven foundation for next-generation training pipelines for frontier reasoning LLMs.
| Subjects: | Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Performance (cs.PF) |
| Cite as: | arXiv:2611.09207 [cs.DC] |
| (or arXiv:2611.09207v1 [cs.DC] for this version) | |
| https://doi.org/10.48550/arXiv.2611.09207 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Robert Gerstenberger [view email]
[v1]
Fri, 6 Nov 2026 09:28:49 UTC (1,289 KB)
来源：arXiv cs.CL · arxiv.org