arXiv cs.AI· Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang·· 1 天前AI 评分35
SO-Arena：评测 System One 模型能力谱系与决策可靠性的基准
SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models
AI 导读
研究者推出 SO-Arena 基准，覆盖 11 个领域和五类能力，从能力谱系与决策可靠性两个维度评测 System One 模型。除准确率外，它还考察选项集受控变化下的决策稳定性和无有效选项时的缺失答案识别能力。对 21 个模型的评测显示，规模扩展与额外推理的收益因领域差异明显，整体表现最强的 System One 模型在可靠性上仅排第七。
正文 · 原文
Abstract:System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.
| Comments: | 32 pages, 10 figures, 7 tables. Project page: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.15777 [cs.CL] |
| (or arXiv:2611.15777v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15777 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lichen Li [view email]
[v1]
Wed, 11 Nov 2026 17:53:45 UTC (2,899 KB)
来源：arXiv cs.AI · arxiv.org