arXiv cs.AI· Yilang Zhang, Bingcong Li, Niao He, Georgios B. Giannakis·· 1 天前AI 评分34
ANCRe：自适应神经连接重分配实现高效深度扩展
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
AI 导读
研究者提出 ANCRe，一个轻量框架，通过参数化并从数据中学习残差连接布局，自适应重分配残差连接，计算与内存开销低于 1%。理论分析证明残差连接布局会从根本上影响收敛行为，甚至造成收敛速率的指数级差距。在 LLM 预训练、扩散模型和深层 ResNet 上的实验显示，相比传统残差连接，ANCRe 能持续加速收敛、提升性能并增强深度效率。
正文 · 原文
Abstract:Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This paper revisits the default mechanism for deepening neural networks, namely residual connections, from an optimization perspective. Rigorous analysis proves that the layout of residual connections can fundamentally shape convergence behavior, and even induces an exponential gap in convergence rates. Prompted by this insight, we introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data. ANCRe adaptively reassigns residual connections with negligible computational and memory overhead ($<1\%$), while enabling more effective utilization of network depth. Extensive numerical tests across pre-training of large language models, diffusion models, and deep ResNets demonstrate consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.09009 [cs.LG] |
| (or arXiv:2602.09009v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.09009 arXiv-issued DOI via DataCite |
|
| Journal reference: | Advances in Neural Information Processing Systems, 2026 |
Submission history
From: Yilang Zhang [view email]
[v1]
Mon, 9 Feb 2026 18:54:18 UTC (5,679 KB)
[v2]
Thu, 1 Oct 2026 21:55:31 UTC (5,686 KB)
[v3]
Tue, 6 Oct 2026 19:52:15 UTC (5,686 KB)
[v4]
Tue, 13 Oct 2026 21:55:31 UTC (5,689 KB)
[v5]
Wed, 11 Nov 2026 22:03:55 UTC (5,689 KB)
来源：arXiv cs.AI · arxiv.org