跳到正文
arXiv cs.CL· Xinying Hu, You Wu, Haoyi Wu, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Kewei Tu·· 23 小时前AI 评分22

TPHC:用张量并行分组更新改进 Hyper-Connections

TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates

AI 导读

TPHC 将张量并行(TP)分片与 Hyper-Connections 的残差流结合,让每个 TP 分片接收独立的残差流混合、写回时保持各分片输出分离,在不增加注意力或 FFN 子层参数与 FLOPs 的前提下提升残差流更新的表达能力。

正文 · 原文

View PDF HTML (experimental)

Abstract:Hyper-Connections (HC) expand the residual stream of a Transformer into $N$ parallel streams, and manifold-constrained HC (mHC) stabilizes them at scale. However, mHC benefits little from more than four streams, partly because each sublayer writes a single output to all streams, limiting how differently they can be updated. In this work, we propose TPHC, which integrates tensor-parallel (TP) shards with HC streams to promote greater independence across streams. Specifically, we route a separate mixture of the residual streams to each TP shard and keep the individual shard outputs separate during write-back, so each stream can receive a different mixture of those outputs. This design makes the residual-stream updates more expressive without adding parameters or FLOPs to the attention or FFN sublayers. With native support for stream sharding under TP, TPHC requires only 9% more training time than mHC on an 8B model when both use four streams. Increasing the number of streams from four to sixteen raises its training time by only 19%. On a 1.1B model trained on 100B tokens, TPHC reduces validation loss by 0.7-1.0% and raises average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3-0.7 percentage points relative to mHC at matched stream counts $N\in\{4,8,16\}$. Our code is available at this https URL.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2611.23026 [cs.CL]
  (or arXiv:2611.23026v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.23026

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: You Wu [view email]
[v1] Tue, 17 Nov 2026 16:31:38 UTC (57 KB)

来源:arXiv cs.CL · arxiv.org