arXiv cs.CL· Tianxiang Dai, Jonathan Fan·· 10 小时前AI 评分48
语言模型任务建模的最小探针:205 个模型的计数能力测试
A minimal probe of task modeling in language models
AI 导读
研究者提出用"计数相同项目"作为检验语言模型任务建模能力的最简测试,并定义计数容量(CC)为模型能准确计数的最长序列长度。在 205 个模型上测得 CC 中位数为 50 项、最高 627 项,而各年发布模型的中位 CC 仅提升 1.6 倍,同期上下文窗口扩大约 80 倍。在 77 组配对中推理并未改变中位 CC,且 CC 在控制通用能力后与 METR 任务时长相关。
正文 · 原文
Abstract:Language models read long contexts and score well on knowledge and reasoning benchmarks, but neither shows whether a model can do work whose next step depends on everything that came before, as a long-running agent must. We call this ability task modeling and propose counting identical items as its simplest test: every item adds to the total and no item records it. We define counting capacity (CC) as the sequence length up to which a model returns the exact count more often than not, and measure it for 205 models. Every resolved measurement loses count abruptly, at a median of 50 items and at most 627; across release years the median rose 1.6-fold while context windows grew about 80-fold. Changing the counted symbol or the delimiter moves a model's CC severalfold, yet the model accounts for about three quarters of the variance across 16 models and 11 encodings. Reasoning leaves the median CC unchanged across 77 matched pairs. In six open-weight models the count stays readable in early layers past the boundary but does not reach the output, and an activation edit that carries the count forward also carries the state of an object-tracking task. CC tracks the METR task horizon after controlling for general capability, and rises with capacity on grid-execution tasks and, with reasoning off, on object tracking. We do not show that counting is necessary or sufficient for long-horizon work; that remains a hypothesis, which this cheap measurement can test.
| Comments: | for accessing the data and code for reproduction of the study, see this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.02028 [cs.CL] |
| (or arXiv:2605.02028v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.02028 arXiv-issued DOI via DataCite |
Submission history
From: Tianxiang Dai [view email]
[v1]
Sun, 3 May 2026 19:27:36 UTC (2,893 KB)
[v2]
Sat, 16 May 2026 23:34:26 UTC (3,709 KB)
[v3]
Wed, 18 Nov 2026 18:44:26 UTC (1,841 KB)
来源:arXiv cs.CL · arxiv.org