arXiv cs.CL· Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsaecker·· 13 小时前AI 评分58
研究用国际象棋区分大语言模型的泛化与记忆
Disentangling generalization and memorization in large language models using chess
AI 导读
研究者提出以国际象棋作为受控测试平台，按相关先验的密度对局面分类，从而在不需要了解模型训练数据的情况下区分记忆与泛化。对GPT系列的长时段分析和Claude Opus、Gemini等当代模型的评测显示，先验越稀疏性能越差，在少先验任务上基座模型回落到随机走子水平，各模型提出非法走子的比例也明显上升。推理增强能提升表现，但每个token的相对边际收益随先验减少而下降。
正文 · 原文
Abstract:Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall or genuine reasoning ability. We introduce chess as a controlled testbed aimed at disentangling these faculties. Leveraging the game's structure and scalable engine evaluations, we construct a taxonomy of positions varying in density of relevant priors -- ranging from common states solvable by memorization to completely novel ones requiring generalization. Crucially, our approach achieves this distinction without requiring explicit knowledge of the models' training data. Applying this taxonomy, we combine a longitudinal analysis of the GPT lineage with a rigorous evaluation of contemporary models, including Claude Opus and Gemini. Our analysis reveals a steep gradient: performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline, and all models propose illegal moves at sharply increasing rates, indicating that they fail to reliably apply the fixed rules of the game in unfamiliar positions. While newer models improve, progress slows significantly for tasks with sparse priors. Furthermore, while reasoning-augmented inference improves performance, its relative marginal benefit per token decreases in the absence of relevant priors. These results suggest limitations in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust performance when deprived of relevant priors.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.16823 [cs.CL] |
| (or arXiv:2601.16823v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.16823 arXiv-issued DOI via DataCite |
Submission history
From: Leonard S. Pleiss [view email]
[v1]
Fri, 23 Jan 2026 15:23:08 UTC (537 KB)
[v2]
Tue, 19 May 2026 12:03:44 UTC (698 KB)
[v3]
Tue, 20 Oct 2026 21:13:38 UTC (700 KB)
来源：arXiv cs.CL · arxiv.org