arXiv cs.CL· Dengzhe Hou, Kazunori D Yamada·· 11 小时前AI 评分34
S1CogBench：System One 决策模型在认知实验中能做什么、不能做什么
S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments
AI 导读
研究者提出 S1CogBench，用 24 项认知任务评测 System One 决策模型（含 Jev、Laya 及五个开源模型），这是首个针对该类模型的认知实验基准。结果显示决策模型可靠性差异很大，行为表现好并不代表能预测人类，聚合分数会掩盖决策对条件的响应，其速度主要来自不生成文本。
正文 · 原文
Abstract:Language models are no longer used only to generate text. Software also asks them for decisions, in routing, guardrails, and automated gates. System One decision models, also called typed decision models, such as Jev and Laya, return one fast decision over the legal options, leaving no text to inspect. However, a fast, well-formed decision does not show that the model used the evidence. Cognitive experiments can test this, because they vary task demands and measure how behavior changes, yet existing cognitive benchmarks evaluate language and multimodal models. We introduce S1CogBench to ask what these models can and cannot do on 24 cognitive tasks, evaluating five open models and Jev and keeping conditions and scoring apart from disclosed adaptations. We find that decision models vary widely in reliability, acting well does not imply predicting humans, aggregate scores hide how decisions respond to conditions, and speed comes mainly from answering without generation. To our knowledge, S1CogBench is the first benchmark of System One decision models on established cognitive experiments, and it gives a shared test of whether their decisions respond to evidence, feedback, memory load, and future opportunities before software relies on them.
| Comments: | 13 pages, 4 figures, 12 tables. Code at this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Neurons and Cognition (q-bio.NC) |
| Cite as: | arXiv:2611.00911 [cs.CL] |
| (or arXiv:2611.00911v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.00911 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dengzhe Hou [view email]
[v1]
Sat, 31 Oct 2026 13:09:19 UTC (232 KB)
来源：arXiv cs.CL · arxiv.org