arXiv cs.CL· Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali·· 12 小时前AI 评分38
双路径架构：让 LLM 的计算与容量分别扩展
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
AI 导读
研究者提出双路径块，在单层内并行暴露计算与容量两条扩展轴：深层子层以共享参数重复应用 K 次，宽层子层放大前馈网络仅应用一次，由独立的逐 token 门控融合。在两个 FLOP 预算下，最佳双路径配置在语言建模、常识与数学指标上均超过等 FLOP 的单轴对照组，且比宽度扩展基线少用 17-33% 参数。学到的门控呈现系统性分配：虚词与实词倾向宽路径，标点、符号与算术 token 倾向深路径。
正文 · 原文
Abstract:Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We argue that compute, the number of sequential operations applied to a hidden state, and capacity, the parameters available at a single step, are distinct scaling axes that standard transformer layers conflate. We propose a dual-path block that exposes both axes as parallel pathways within a single layer: a deep sublayer re-applied $K$ times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once, combined by independent per-token gates. At two FLOP budgets, the best dual-path configuration surpasses both iso-FLOP single-axis controls on aggregate language modelling, commonsense, and math metrics, while using 17-33% fewer parameters than the width-scaled baseline. The learned gates are directly interpretable and show systematic per-token allocation: function words and lexical content trend wide, while punctuation, symbols, and arithmetic tokens trend deep. Our results suggest that compute and capacity can be scaled separately, with the model balancing the two token by token.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.30202 [cs.CL] |
| (or arXiv:2605.30202v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.30202 arXiv-issued DOI via DataCite |
Submission history
From: Markus Frey [view email]
[v1]
Thu, 28 May 2026 16:41:36 UTC (263 KB)
[v2]
Wed, 4 Nov 2026 13:31:51 UTC (289 KB)
来源：arXiv cs.CL · arxiv.org