arXiv cs.MA· Vartan Shadarevian, Kia Ghods, Alex Kenich, Anany Kotawala·· 11 小时前AI 评分38
GENSTRAT：面向大语言模型战略推理科学的程序化生成评测系统
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
AI 导读
GENSTRAT 通过程序化生成两人零和不完全信息下注游戏来评测 LLM 的战略能力，可随时按需生成全新对局，避免固定游戏套件饱和与污染问题。基于该生成器构建的 50 款游戏基准上，12 个前沿与开源权重模型进行了超 40,000 场 LLM 对 LLM 对局；在 8 款最难游戏上，最佳与最差模型的差距是全套游戏上该差距的 1.7 倍。
正文 · 原文
Abstract:Fixed game suites for evaluating strategic reasoning in large language models (LLMs) saturate and are hard to keep free of contamination. We introduce GENSTRAT, a system for evaluating the strategic capabilities of LLMs driven by procedural generation of two-player zero-sum imperfect-information betting games, allowing an evaluator to draw novel, fresh games on demand. Each game is characterized on a scale of five complexity axes (state space, temporal contingency, opponent dependence, private information, precision) and measured by Monte Carlo simulation. Using this generator, we construct a 50-game benchmark and hold a tournament of twelve frontier and open-weight models, gathering data from over 40,000 LLM vs. LLM matches. We further construct a composite complexity axis, and find that, on the eight hardest games, the gap between the best and worst model is 1.7x its value across the whole set of games. Ablations show that a higher thinking budget, disclosing the opponent's strategy, and higher reasoning matched to a game's specific demands (for private information and planning depth) were associated with higher margins for the LLM player.
| Comments: | 33 pages, 8 figures, 17 tables (3 figures, 2 tables in main paper) |
| Subjects: | Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2605.23238 [cs.AI] |
| (or arXiv:2605.23238v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.23238 arXiv-issued DOI via DataCite |
Submission history
From: Vartan Shadarevian [view email]
[v1]
Fri, 22 May 2026 05:13:45 UTC (1,940 KB)
[v2]
Thu, 29 Oct 2026 23:30:39 UTC (1,122 KB)
来源：arXiv cs.MA · arxiv.org