arXiv cs.CL· Dongyu Gong·· 2 天前AI 评分26
语言模型在工作记忆中共享读出瓶颈
Language Models Share a Readout Bottleneck in Working Memory
AI 导读
研究在 15 个预训练 LLM（70M 至 14B 参数，来自三个模型家族）及从零训练的最小 Transformer 上发现，LLM 在 N-back 任务中表现出与人类相似的工作记忆特征：记忆负载 N 增大时性能下降，错误集中于相邻错误位置的干扰项。
正文 · 原文
Abstract:Large language models (LLMs) reproduce a human working-memory signature on the N-back task: performance declines as memory load N increases, and errors concentrate on lures, items that repeat the current stimulus at a neighboring but incorrect serial position. We ask whether this shared behavioral signature reflects a shared mechanism. The literature offers two candidate accounts. On a precision account, softmax attention cannot concentrate its weight on the item N positions back, so adjacent items are retrieved instead. On an interference account, attention selects the correct item, but the content of competing items corrupts the decision. We dissociate these accounts in 15 pretrained LLMs from three families (70 million to 14 billion parameters) and in minimal transformers trained from scratch. In all three families, the attention heads that carry the task attend to the target on most missed matches. Linear probes decode near-perfectly whether a matching letter occurred at the target or at a lure position, even on trials where the model false-alarms to the lure, although its own decisions barely distinguish the two. Sharpening attention yields negligible gains, whereas suppressing the letter identity of non-target items improves sensitivity (d') in nearly all model-by-load conditions. In larger models the task is spread over more heads, and removing interference helps more. In minimal transformers, precision limits a single attention layer, but a second layer removes this constraint. The working-memory limit of LLMs therefore lies in reading items out under interference rather than in locating them, consistent with interference accounts of human working memory.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Neurons and Cognition (q-bio.NC) |
| Cite as: | arXiv:2610.21176 [cs.LG] |
| (or arXiv:2610.21176v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21176 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongyu Gong [view email]
[v1]
Thu, 15 Oct 2026 03:32:37 UTC (141 KB)
来源：arXiv cs.CL · arxiv.org