arXiv cs.AI· Xinhang Zhang, Boning Zhang, Chengchun Liu, Chunpu Li, Lei Sun, Weifeng Zhang, Limin Xiao·· 12 小时前AI 评分29
CHARM+：面向跨硬件场景的重合并多流注意力机制
CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism
AI 导读
CHARM+ 提出跨硬件注意力机制，通过将顺序通信重组为多个层级并解耦跨硬件与硬件内通信，消除带宽不均衡带来的同步开销。在双 NUMA PCIe 架构上，其前向与反向算法相比 Ring-Attention 平均加速 1.3x 和 1.7x；在双节点 NVLink 架构上分别加速 2.8x 和 2.9x。该工作已被 NeurIPS 2026 接收为 Poster。
正文 · 原文
Abstract:As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation-communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We reorganize sequential communications into several levels and achieve intra-level parallelism and computation overlap. Furthermore, we decouple cross-hardware communication from intra-hardware communication, eliminating synchronization overhead caused by bandwidth imbalance. Forward and backward algorithms were evaluated on multiple hardware architectures and consistently outperformed state-of-the-art methods. Compared to Ring-Attention, our forward and backward algorithms achieve average speedups of 1.3x and 1.7x on a dual-NUMA PCIe architecture, and 2.8x and 2.9x on a dual-node NVLink architecture, respectively.
| Comments: | Accepted to NeurIPS 2026 (Poster) |
| Subjects: | Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2611.05906 [cs.AI] |
| (or arXiv:2611.05906v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05906 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Boning Zhang [view email]
[v1]
Wed, 4 Nov 2026 03:57:56 UTC (758 KB)
来源：arXiv cs.AI · arxiv.org