arXiv cs.AI· Noel Thomas·· 2 天前AI 评分22
ChaosBench-Logic：面向混沌动力系统逻辑与符号推理的基准
ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems
AI 导读
研究者提出 ChaosBench-Logic，用统一一阶逻辑（FOL）本体覆盖 30 个动力系统、11 个语义谓词，生成 621 道题，涵盖多跳蕴含、跨系统类比、反事实推理、偏见探测和多轮对话七类任务。
正文 · 原文
Abstract:Large language models (LLMs) excel at natural language tasks but remain brittle in domains requiring precise logical and symbolic reasoning. Chaotic dynamical systems provide an especially demanding test because chaos is deterministic yet often misinterpreted as randomness or complexity. We introduce ChaosBench-Logic, a benchmark that evaluates LLM reasoning across 30 diverse dynamical systems using a unified first-order logic (FOL) ontology. Each system is annotated with truth assignments for 11 semantic predicates, and 621 questions are generated across seven reasoning categories, including multi-hop implications, cross-system analogies, counterfactual reasoning, bias probes, and multi-turn dialogues. We define metrics for logical accuracy, implication consistency, dialogue coherence, and contradiction, and we release an open-source evaluation pipeline. Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91-94% per-item accuracy, yet still score 0% on compositional items and exhibit fragile global coherence. Dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zero-shot). ChaosBench-Logic provides a rigorous testbed for diagnosing such failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs.
| Comments: | 7 pages, 0 figures, Accepted to AAAI-26 Bridge Program: Logical and Symbolic Reasoning in Language Models (camera-ready). v2: corrected bibliographic errors in the references |
| Subjects: | Artificial Intelligence (cs.AI); Information Theory (cs.IT) |
| Cite as: | arXiv:2601.01982 [cs.AI] |
| (or arXiv:2601.01982v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2601.01982 arXiv-issued DOI via DataCite |
|
| Journal reference: | AAAI 2026 Bridge Program on Logical and Symbolic Reasoning in Language Models, Singapore, Jan 2026 |
Submission history
From: Noel Thomas [view email]
[v1]
Mon, 5 Jan 2026 10:36:40 UTC (24 KB)
[v2]
Wed, 14 Oct 2026 18:40:45 UTC (24 KB)
来源：arXiv cs.AI · arxiv.org