arXiv cs.CL· Dengzhe Hou, Lingyu Jiang, Fangzhou Lin, Kazunori D Yamada·· 11 小时前AI 评分40
CogBattery：语言模型认知任务行为与结构的人类参照评测
CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models
AI 导读
CogBattery 用 13 个认知范式的文本改编版评测 54 个开源权重语言模型，并与各原始研究中的人类数据逐项对比。模型和人类一样对 n-back 诱饵反应更强，但在普通非目标上误报远多于人类；多数模型的置信度区分正误的能力弱于人类，且无论答对与否都选同一赌注。模型得分主要沿两个维度分布，第一维与规模密切相关，但更大模型并未更接近人类，预设认知分组也不优于同规模随机分组。
正文 · 原文
Abstract:Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data from its source study. Like humans, models respond more to n-back lures than to controls, yet they make far more false alarms on ordinary non-targets. Most models' reported confidence separates correct from incorrect answers less well than humans' does, and most always choose the same wager, including models that classify accurately. These patterns span many model families. Across tasks, model scores share two dimensions. The first is closely tied to scale, yet larger models are not consistently closer to humans. Predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size. These results show that matching human effect directions does not establish human-like behavior. CogBattery provides condition-level human references for testing errors and confidence reports in new models.
| Comments: | 13 pages, 3 figures, 5 tables. Code at this https URL. Renamed from CogArena to avoid confusion with an unrelated benchmark of the same name |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.24999 [cs.CL] |
| (or arXiv:2607.24999v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.24999 arXiv-issued DOI via DataCite |
Submission history
From: Dengzhe Hou [view email]
[v1]
Mon, 27 Jul 2026 18:56:13 UTC (957 KB)
[v2]
Fri, 30 Oct 2026 13:55:34 UTC (855 KB)
来源：arXiv cs.CL · arxiv.org