跳到正文
arXiv cs.AI· Alexander Boesgaard Lorup·· 10 小时前AI 评分38

Openhagen Worlds:用于测量与训练科学发现智能体的校准环境

Openhagen Worlds: Calibrated Environments for Measuring and Training Scientific Discovery Agents

AI 导读

Openhagen Worlds 推出 27 个程序化生成环境、178 个校准难度等级,用于测量和训练科学发现智能体,每个等级需脚本 oracle 在 64 个开发实例上成功率至少 0.75 才被收录。

正文 · 原文

View PDF HTML (experimental)

Abstract:Agents that do science must choose informative experiments and reason from their results, but most discovery benchmarks cannot separate that skill from recall, unstated scoring rules or shortcut policies. We introduce Openhagen Worlds, a suite of 27 procedurally generated environments (some with finite truth spaces), from classic puzzles of experimental reasoning to simulator-grounded physics and chemistry, in which an agent runs experiments under a budget and commits an answer that a deterministic verifier checks against a hidden truth. Every observation states the complete scoring rule, and each of the 178 difficulty levels is calibrated: it is admitted only if, on 64 development instances, a scripted oracle succeeds on at least 0.75, the strongest plausible non-deductive policy on at most 0.25 and each gated shortcut on at most 0.35. The 92 core levels span all 27 families and are sized to train small models; the 86 frontier rungs extend the 15 discovery worlds with harder questions and with campaigns of three, five and eight or more committed stages on one hidden system. On the same 84 withheld frontier instances, one per family and rung, DeepSeek V4.1 Flash solves 32 at its default reasoning effort with 16,384 tokens per call and 57 at its maximum effort with up to 393,216 tokens per call (oracle: 79); the headroom that remains is largest in eight-stage campaigns. On-policy expert iteration gives a 9B model modest gains: its core-tier success on trained families rises from 0.090 to 0.148, partly through better action format, with no detectable loss on GPQA Diamond. A hosted service plays the core tier's withheld instances.
Comments: 41 pages. Ancillary files: a public sample of 48 development instances with scripted-oracle episodes. Hosted evaluation: this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2611.24711 [cs.LG]
  (or arXiv:2611.24711v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.24711

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Alexander Lorup [view email]
[v1] Wed, 18 Nov 2026 16:36:18 UTC (326 KB)

来源:arXiv cs.AI · arxiv.org