arXiv cs.AI· Yangsheng Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tang, Junchi Zeng, Mingji Han, Ray Ying, Key Zhang, Luo Mai·· 12 小时前AI 评分34
HyScale：为学习型稀疏注意力扩展长上下文训练
HyScale: Scaling Long-Context Training for Learned Sparse Attention
AI 导读
HyScale 是面向学习型稀疏注意力的长上下文训练系统，通过 HyTopK 与 selection replay 消除完整分数表和重复选择开销，在 512 张 H20 GPU 上将 GLM-5.1 的 1M token 序列训练加速 2.07 倍（对比 Megatron-LM）。
正文 · 原文
Authors:Yangsheng Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tang, Junchi Zeng, Mingji Han, Ray Ying, Key Zhang, Luo Mai
Abstract:We present HyScale, a long-context training system for learned sparse attention. Learned sparse attention is a promising architecture for million-token contexts, widely adopted in production since DeepSeek-V3.2. Selection, however, grows into the training bottleneck. For each query, it scores every previous token whereas sparse attention reads only $k$; worse, current systems materialize a full score table and repeat selection during recomputation. Our key insight is that the output of selection is not an activation, but rather a set of indices. Building on it, HyScale introduces HyTopK and selection replay. HyTopK finds each query's exact top-$k$ on the fly without a score table, while selection replay reuses forward-pass selection in recomputation. Together with holistic system optimizations tackling end-to-end training challenges, HyScale speeds up GLM-5.1 training on 1M-token sequences by 2.07$\times$ over Megatron-LM on 512 H20 GPUs. It also speeds up the CSA layers of DeepSeek-V4-Flash and the QSA layers of Qwen3.8-Flash-Next by 2.01$\times$ and 1.57$\times$ at 1M tokens on 32 H800 GPUs.
| Subjects: | Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2610.32057 [cs.AI] |
| (or arXiv:2610.32057v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32057 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yangshen Deng [view email]
[v1]
Fri, 23 Oct 2026 01:54:02 UTC (735 KB)
来源：arXiv cs.AI · arxiv.org