arXiv cs.AI· Mostafa 'Neo' Mohsenvand·· 4 小时前AI 评分58
Attention at Absolute Zero:零温度读取 witness 的高阶注意力层
Attention at Absolute Zero
AI 导读
Mostafa 'Neo' Mohsenvand 提出一种高阶注意力层,其状态建立在成对条目上,以零温度读取 witness,只保留最优项及并列计数,并仅从答案训练。
正文 · 原文
Abstract:Attention scores candidate supports of an answer, which we call witnesses, and reads the scores at a temperature. We build a higher-order attention layer whose state lives on pairs of items: for a pair (a, b), each third item k is a witness judged through its legs (a, k) and (k, b). It reads witnesses at zero temperature, keeping the best and the number tied with it, passes on only bits comparing such counts, rebuilds its pair state from the input and these bits at every loop, and is trained from answers alone. Each choice removes one way soft attention fails at size: dispersed weight, unresolved gaps between learned scores, and a drifting soft state. In maximum retrieval it is right on at least 99.95% of sets 1,000 times larger than in training, on all ten seeds; trained with a soft count, the same network falls to 42% at 4,096 items. On CLRS-30 without hints, a general form of the layer (half its channels read at zero temperature, numbers embedded, state carried) beats the best published model, which uses hints on some tasks; the full cold layer is exact on every test input of 9 of the 12 tasks whose answers depend on numbers only through comparisons. From yes-or-no answers it decides 2-SAT and two-colouring exactly at 500 variables and, as our theory predicts, not parity equations. Beside a frozen 7B language model it chains family relations exactly at 1,000 statements after training on 16, where the model alone falls to 44%; finite checks on the weights prove every block that learned correct at every size, on acyclic contexts and given what the model reads. On CLUTRR it is right on every story of five to ten relations given the graphs, learning the 62 rules exactly; from text, with the model frozen, no hand-written rule and only a small read-in trained, it lifts an open 7B model from 33% to 65%, above every system we found without hand-written rules except frontier reasoning models.
| Comments: | 51 pages, 9 figures, 1 Extended Data figure, 4 supplementary tables. Replaces v1, "Attention Is Only as Good as Its Witnesses". Code (Apache 2.0): this https URL ; demo: this https URL ; models: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE) |
| Cite as: | arXiv:2611.11937 [cs.LG] |
| (or arXiv:2611.11937v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.11937 arXiv-issued DOI via DataCite |
Submission history
From: Mostafa 'Neo' Mohsenvand [view email]
[v1]
Mon, 9 Nov 2026 06:08:10 UTC (1,792 KB)
[v2]
Tue, 17 Nov 2026 14:14:46 UTC (1,719 KB)
来源:arXiv cs.AI · arxiv.org