arXiv cs.AI· Noel Thomas·· 2 天前AI 评分24
ChaosBench-Logic v2：大规模评估 LLM 对动力系统的逻辑推理
ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale
AI 导读
ChaosBench-Logic v2 是一个包含 40,886 道题、覆盖 165 个动力系统、27 个 FOL 谓词与 78 条公理边的基准，并配套 CARE 评估协议。
正文 · 原文
Abstract:Standard accuracy on binary reasoning benchmarks hides critical failure modes: prior collapse, inconsistency under paraphrase, and inability to reason about parameter-dependent dynamics. We present ChaosBench-Logic v2, a 40,886-question benchmark over 165 dynamical systems with 27 FOL predicates and 78 axiom edges, together with CARE (Calibration- and Adversarial-Robust Evaluation), a protocol that surfaces these pathologies. Evaluating 14 models, we find that regime-transition reasoning remains near random (MCC = 0.05) even for frontier models, whereas FOL deduction with given premises reaches MCC = 0.52. Per-family decomposition shows that the proprietary-model advantage concentrates on cross-indicator (+0.40) and consistency tasks, while open-source Qwen 2.5-32B dominates indicator diagnostics (0.91 vs. 0.45). Two models exhibit negative MCC on bifurcation questions, confirmed as systematic anti-correlation via confusion-matrix analysis.
| Comments: | 14 pages, 8 figures. Published at the ICLR 2026 Workshop on LLM Reasoning. v2: corrected bibliographic errors in the references |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.24305 [cs.LG] |
| (or arXiv:2605.24305v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.24305 arXiv-issued DOI via DataCite |
Submission history
From: Noel Thomas [view email]
[v1]
Sat, 23 May 2026 00:34:17 UTC (172 KB)
[v2]
Wed, 14 Oct 2026 18:41:06 UTC (172 KB)
来源：arXiv cs.AI · arxiv.org