arXiv cs.CL· Tsung-Han Wu, Minsuk Heo, Kenny Tan·· 12 小时前AI 评分31
循环语言模型中的 halting head 真能省算力吗？研究揭示两个问题
Do Halting Heads Really Save Compute in Looped Language Models?
AI 导读
从头预训练最多 1.3B 参数的循环语言模型后发现，halting head 存在两个问题：在全部迭代混合训练下，模型预测 token 所需迭代数不到一半，但按标准停止规则几乎每个 token 仍跑满所有迭代，因为训练从未对 token 实际停止的那一次迭代打分；改为对每次迭代打分可修复，但需强惩罚偏向早停。
正文 · 原文
Abstract:Looped language models run each token through the same layers several times, and each run is one iteration. To save compute, an extra layer, the halting head, gives after each iteration the probability that the token stops there, so that easy tokens can stop early. Pretraining models from scratch, up to 1.3B parameters, we find two problems. First, heads trained on a mix of all iterations predict that tokens need under half the iterations, yet under the standard stopping rule tokens run almost every iteration, because this training never scores the single iteration where a token stops. Scoring every iteration instead fixes this, but in our runs it works only with a strong penalty favoring early stops. Second, savings are commonly reported as an average number of iterations per token, but when sequences are generated in batches that run until their slowest token stops, a model without a head is almost as good at that cost. The reported average can therefore overstate the saving. We recommend reporting the iterations tokens actually run, with the stopping threshold and batch size, and give a test, run before training a head, that bounds its gain.
| Comments: | 33 pages, 6 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.32246 [cs.CL] |
| (or arXiv:2610.32246v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32246 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tsung-Han Wu [view email]
[v1]
Fri, 23 Oct 2026 06:56:48 UTC (189 KB)
来源：arXiv cs.CL · arxiv.org