arXiv cs.AI· Jianxin Gao, Beini Hu, Runze Li, Hongjian Fang, Wanli Peng·· 12 小时前AI 评分34
MirrorCraft：Minecraft 隐藏规则变化下的配对评测基准
MirrorCraft: Paired Evaluation under Hidden Rule Changes in Minecraft
AI 导读
MirrorCraft 是一个在 Minecraft 隐藏规则变化下评测 LLM 智能体的配对基准，每个 Mirror 世界复制对应的 Vanilla 世界，仅通过数据包静默修改服务端规则，地形、出生点、资源分布、任务、接口与动作预算保持匹配。
正文 · 原文
Abstract:With the rapid progress of large language models (LLMs), an interesting question has emerged: how do LLM-based agents perform in Minecraft? To date, most Minecraft benchmarks evaluate them under fixed game mechanics, which are extensively documented online. High performance in these settings does not show whether an agent can continue making progress when familiar recipes, drops, and other rules change. In this paper, we introduce MirrorCraft, a paired benchmark for evaluating agents under hidden rule changes in Minecraft. Each Mirror world is a copy of its paired Vanilla world, with selected server-side rules silently modified by the datapack of one rule suite. Terrain, spawn, resource placement, task, interface, and action budget remain matched within every Vanilla-Mirror pair. MirrorCraft includes 10 Vanilla worlds in five controlled biomes, six rule suites, and three progression tasks, and we evaluate two LLMs and six agent configurations under a shared Mineflayer interface. We measure progress with server-verified advancement milestones and completion with success rate, and define the Rule Intervention Effect (RIE) as the performance change between paired Vanilla and Mirror worlds. Experiments with 8,640 episodes show that hidden rule changes have sharply different effects across suites. Among the configurations evaluated without rule descriptions, ReAct, Reflexion, and Voyager achieve the highest pooled Mirror Scores, but ReAct loses the most under compressed resource drops. Providing ReAct with the exact rules yields modest gains in average progress and completion across all three tasks, yet most of the difference between suites remains once the rules are known. MirrorCraft extends Minecraft evaluation beyond fixed mechanics and provides a controlled setting for studying how agents use gameplay outcomes when the rules of the current world differ from familiar ones.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.29218 [cs.AI] |
| (or arXiv:2607.29218v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.29218 arXiv-issued DOI via DataCite |
Submission history
From: Jianxin Gao [view email]
[v1]
Fri, 31 Jul 2026 09:43:54 UTC (9,777 KB)
[v2]
Fri, 23 Oct 2026 15:19:06 UTC (1,410 KB)
来源：arXiv cs.AI · arxiv.org