arXiv cs.CL· Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting·· 11 小时前AI 评分38
Dreamer：深度循环注意力混合框架让潜在推理更高效
Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves
AI 导读
研究者提出深度循环注意力混合框架 Dreamer，将序列注意力、深度注意力与稀疏专家注意力结合，通过沿深度方向的注意力缓解隐藏维度瓶颈。在语言推理基准上，达到与 FLOP、参数和内存匹配的 SOTA 相同准确率所需训练 token 减少 2 至 8 倍，相同训练 token 下性能超过约 2 倍大的 SOTA 模型。其专家选择多样性比 SOTA MoE 高 2 至 11 倍。
正文 · 原文
Abstract:Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes depth-recurrence due to partially fixed layer stacks, and ignores the bottleneck of constant hidden-sizes that restricts many-step latent reasoning. To address this, we introduce a modular framework of depth-recurrent attention mixtures (Dreamer), combining sequence attention, depth attention, and sparse expert attention. It alleviates the hidden-size bottleneck through attention along depth, decouples scaling dimensions, and allows depth-recurrent models to scale efficiently and effectively. Across language reasoning benchmarks, our models require 2 to 8x fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched SOTA, and outperform ca. 2x larger SOTA models with the same training tokens. We further present insights into knowledge usage across depths, e.g., showing 2 to 11x larger expert selection diversity than SOTA MoEs.
| Comments: | NeurIPS 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2601.21582 [cs.AI] |
| (or arXiv:2601.21582v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2601.21582 arXiv-issued DOI via DataCite |
Submission history
From: Jonas Knupp [view email]
[v1]
Thu, 29 Jan 2026 11:44:38 UTC (968 KB)
[v2]
Thu, 5 Nov 2026 10:23:39 UTC (271 KB)
来源：arXiv cs.CL · arxiv.org