arXiv cs.AI· Tidor-Vlad Pricope·· 14 小时前AI 评分38
RetroChessBench：评估大模型重建并证明国际象棋历史的基准
RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories
AI 导读
研究团队推出 RetroChessBench，一个包含 12 项任务的基准，用于检验模型在回溯国际象棋中重建合法棋史、区分可能事件与必然事件的能力。GPT 6 Astra High 在无工具时 24 次尝试中有 22 次未能给出要求的最终答案，配合代码执行时仍有 18 次失败，平均终端 RL 奖励分别为 14.75% 和 27.19%。
正文 · 原文
Abstract:Frontier language models such as GPT 6 Astra have contributed to research mathematics, demonstrating substantial reasoning capabilities. Yet serious reasoning failures persist even in tasks governed by explicit rules. We demonstrate this in retrograde chess which is a playground for pure reasoning, where solvers reconstruct legal histories and distinguish possible events from forced ones. We introduce RetroChessBench, a 12-task benchmark covering exact-length constructions, unique endings, and exhaustive answer sets, with references, rubrics, and a terminal-reward environment. In two trials per task and condition, GPT 6 Astra High fails to produce the required final answer in 22/24 attempts without tools and 18/24 with code execution; mean terminal RL rewards are 14.75% and 27.19%, respectively. The examples identify faulty deductions: failed responses reject legal histories and treat temporary obstructions as permanent. Online retrograde archives often provide puzzles and final move sequences without detailed derivations. We hypothesize that limited exposure to these missing explanations in web-derived training data contributes to the observed reasoning failures. This questions how far frontier mathematical reasoning really generalizes beyond familiar solution patterns. Claims of general intelligence require evidence that models can transfer learnt principles to areas with few worked examples.
| Comments: | 22 pages, 2 figures. Benchmark and RL environment: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39341 [cs.LG] |
| (or arXiv:2610.39341v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39341 arXiv-issued DOI via DataCite |
Submission history
From: Tidor-Vlad Pricope [view email]
[v1]
Thu, 29 Oct 2026 04:32:11 UTC (51 KB)
来源：arXiv cs.AI · arxiv.org