arXiv cs.CY· Daniel S. Epstein·· 12 小时前AI 评分34
借鉴传统兵棋裁决方法改进 AI 评判与评分
Thumb on the scale: Using traditional wargaming adjudication practices to improve AI-judged evaluations and grading
AI 导读
研究将 11 项兵棋推演实践映射到 AI 评估失效场景，并在 9 局 AI 裁判推演、378 个玩家行动上测试裁决方法：同一骰点下，Opus 5 与 Gemini 3.1 Pro 在 Semi-Rigid 模式下重跑会改变 3% 至 6% 的结果，Free 模式下为 14% 至 16%。
正文 · 原文
Abstract:Open-ended AI work has no single correct answer to check, so a judge, often another AI, decides each result. Research on these judges asks which AI to use and how to word the rubric. It rarely tests the procedure that the judge follows. Wargames have no correct answer either, and for two centuries the field has designed that procedure: named methods that set how much freedom the referee has, their known failures, and fixes. We map eleven wargaming practices onto AI evaluation failures. We test one, the adjudication method, on 378 player actions from nine games of an AI-refereed exercise. A player states its odds, the adjudicator turns them into a target number, and a 20-sided die decides pass or fail. With rolls fixed, the target came from the rulebook (Rigid), a judge AI adjusting it (Semi-Rigid), a judge AI setting it alone (Free) or a panel of four judge AIs. In evaluation terms these are a fixed scoring rule, a rubric the judge may adjust, holistic grading and a jury. The judge AIs were Opus 5 and Gemini 3.1 Pro, each re-run. An outcome changes when an action on the same roll passes under one ruling and fails under another. Re-run, a judge changed 3 to 6 per cent of outcomes under Semi-Rigid and 14 to 16 per cent under Free. Swapping Opus for Gemini changed about 3 percentage points more than a re-run did, a gap nine games cannot confirm. The panel changed 40 of the rulebook's 378 outcomes, but the changes cancelled and the pass count moved by 8. A pass rate hides them. One published finding rests on 15 rulings, and most of our AI adjudicators and every re-run changed how often it appeared. We measure how much rulings move, not which ruling is right. The method set how often a judge disagreed with itself. Choose it before the judge AI, report how often the judge disagrees with itself, and count the rulings behind each conclusion. We propose an adjudication statement.
| Comments: | 35 pages, 2 figures. Preprint, version 1. Not peer reviewed |
| Subjects: | Computers and Society (cs.CY) |
| ACM classes: | I.2.0 |
| Cite as: | arXiv:2610.32066 [cs.CY] |
| (or arXiv:2610.32066v1 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32066 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Daniel Epstein [view email]
[v1]
Fri, 23 Oct 2026 02:11:01 UTC (780 KB)
来源：arXiv cs.CY · arxiv.org