arXiv cs.AI· Alina Avanesyan, Alexey Boldyrev, Liqun Deng, Denis Derkach, Mary Godunova, Mikhail Hushchyn, Dmitry Ilvovsky, Irina Knyazeva, Irina Piontkovskaya, Dmitry Polubotko, Fedor Ratnikov, Ekaterina Trofimova, Zhenhe Zhang·· 10 小时前AI 评分52
STEMBench 发布:用各领域评价标准评测 LLM 的博士后级自然科学问题能力
Benchmarking LLM abilities on postdoc-level natural science problems
AI 导读
研究者提出 STEMBench,用高能物理、天体物理和神经科学三个领域的 12 个研究问题评测 LLM,每个问题按该领域自身的评价标准打分,并配有“无领域知识”和“复现已发表方法的专家参考”两个实测锚点。
正文 · 原文
Authors:Alina Avanesyan, Alexey Boldyrev, Liqun Deng, Denis Derkach, Mary Godunova, Mikhail Hushchyn, Dmitry Ilvovsky, Irina Knyazeva, Irina Piontkovskaya, Dmitry Polubotko, Fedor Ratnikov, Ekaterina Trofimova, Zhenhe Zhang
Abstract:Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.
| Comments: | 22 pages, 7 tables, 2 figures |
| Subjects: | Artificial Intelligence (cs.AI); Data Analysis, Statistics and Probability (physics.data-an) |
| Cite as: | arXiv:2611.24321 [cs.AI] |
| (or arXiv:2611.24321v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24321 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Denis Derkach [view email]
[v1]
Wed, 18 Nov 2026 12:25:33 UTC (264 KB)
来源:arXiv cs.AI · arxiv.org