arXiv cs.CL· Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, Bizhao Shi, Zhuo Xia, Yebin Yang, Yizhou Chen, Yuliang Liu, Yu Cheng, Junchi Yan·· 11 小时前AI 评分38
Hyper-Connections 架构设计研究：简化、稳定性与可扩展性
On the Design of Hyper-Connections: Simplification, Stability, and Scalability
AI 导读
研究系统分析了 Hyper-Connections（HC）架构的组件，提出简化设计 SimpleHC：移除残差混合，并通过共享内存访问简化执行。SimpleHC 在最大测试规模下相比 vanilla 残差基线仅增加 2.4% 端到端训练时间，在 15B 到 30B MoE 模型上保持与 mHC 相当的性能，梯度动态更平滑且在压力下保持稳定。
正文 · 原文
Authors:Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, Bizhao Shi, Zhuo Xia, Yebin Yang, Yizhou Chen, Yuliang Liu, Yu Cheng, Junchi Yan
Abstract:Hyper-Connections (HC) architectures are increasingly adopted in frontier language models, yet their components remain insufficiently understood and introduce non-negligible training overhead. In this work, we systematically study the HC architecture to understand its components and identify simpler, more efficient designs. We find that learned read/write gating without mixing and unconstrained mixing with fixed gates form two effective interaction groups: either alone achieves performance comparable to mHC. We also identify repeated access to the expanded residual state as a major source of training overhead, even without mixing. These findings lead to SimpleHC, which simplifies the architecture by removing residual mixing and simplifies execution through shared memory access, using the resulting additive structure and shared gate inputs. Layer-specific gates share a group-entry state, enabling batched reads and deferred writes; scalar corrections exactly preserve preceding sublayer output contributions. Our analyses support both the retained residual structure and the design of shared memory access. SimpleHC incurs only 2.4% extra end-to-end training time over a vanilla residual baseline at the largest tested scale. It maintains performance comparable to mHC when scaling from 15B to 30B MoE models, exhibits smoother gradient dynamics during standard training, and remains stable under stress. We hope SimpleHC will encourage a broader pursuit of simplicity in foundation model design, making performance gains more accessible and practical at scale.
| Comments: | Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05984 [cs.LG] |
| (or arXiv:2611.05984v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05984 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiangdong Zhang [view email]
[v1]
Wed, 4 Nov 2026 05:22:05 UTC (9,408 KB)
来源：arXiv cs.CL · arxiv.org