九个 AI 智能体挑战美国国家实验室开放研究问题，无一达到专家标准

According to the paper, none of the 135 submissions from nine current AI agents met all requirements. The authors see the benchmark as a testing ground to systematically improve AI agents for research, not to question whether AI belongs in science at all.

A total of 114 submissions already fail the integrity check: most claim work that was never done or cite numbers that contradict their own logs. The agents readily produced texts that look like follow-up studies, the authors write, but none met the experts' standards.

Diagram showing a researcher figure who defines a follow-up direction to the LCHS algorithm along with success criteria. Next to it are set diagrams of possible follow-up work and two evaluation rubrics, one for research direction and one for integrity. — Experts define the research direction and minimum requirements for each task, which then become rubrics for quality and honesty. | Image: Kim et al.

Experts set the bar

Each agent receives a scientific paper along with its code and data, plus a research direction. Some of these papers have already been published, while others have only been submitted. From there, the agent is supposed to carry out a follow-up study and write it up as a paper. The 15 tasks come from ten researchers across quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. The directions were proposed by the original papers' authors themselves, and to their knowledge nothing has been published on these topics yet.

A timeline of four publications on LCU and LCHS between 2012 and 2026 at the top, followed by the question "Can an AI agent extend this line?" Below it, a workspace containing a task description, the original paper, code, and data, from which an AI agent produces follow-up papers, code, and logs. — Each agent receives an original paper with its code and data and is expected to produce an independent follow-up study. | Image: Kim et al.

Open research poses a basic problem: there's no answer key. AI reviewers, which have often been used for this purpose, are considered unreliable. So the experts defined the minimum conditions a successful follow-up study has to meet.

The evaluation checks whether a submission is honest, whether it's complete, and whether it clears the experts' bar. GPT-6 Astra handles the scoring, and each submission is judged four times. The agents tested were Claude Opus 4.8, Claude Opus 5, Claude Fable 5, Claude Opus 5.5, GPT-5.6 Sol, GPT-6 Astra, Deepseek V4 Pro, Kimi K3, and GLM-5.3. They had access to powerful machines, internet connectivity, no time limit, and up to five restarts to improve their work. In total, they ran for nearly 2,000 hours.

Only three agents produce honest work at all

Just three agents managed submissions without integrity violations: GPT-6 Astra on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every submission from the remaining six agents contained at least one violation.

The most common problems are mismatches between paper and code, contradictory numbers, and false claims about experiments that were supposedly carried out. Misreported baseline values pile up among the open models Deepseek V4 Pro, Kimi K3, and GLM-5.3. Two submissions with hardcoded rather than computed results came from Kimi K3 and GLM-5.3.

Results table from the "Into the Unknown" benchmark for nine models ranging from Claude Opus 4.8 to GLM-5.3. It shows pass rates at the individual checkpoint level as well as loose and strict paper-level scoring. In the column for all criteria combined, every model scores 0.00 percent. — GPT-6 Astra scores the highest partial marks, but none of the 135 submissions meets all three criteria at once. | Image: Kim et al.

Only one submission is complete, and it came from GPT-6 Astra. Nearly all others leave out required documents, such as a pre-registered experiment plan, and no agent clears the experts' bar. Under a more lenient scoring that accepts 90 percent of the rubric items, GPT-6 Astra comes closest, followed by Claude Opus 5.5 and GPT-5.6 Sol. The open models perform considerably worse overall.

Two heatmaps showing how often submissions from nine models fail specific integrity and direction criteria, such as contradictory numbers, mismatches between paper and code, or missing documentation. — Most integrity violations stem from numbers that contradict the agents' own logs and from mismatches between paper and code. | Image: Kim et al.

Verification script confirms itself

The researchers analyzed 64 work traces and found three typical failure modes. The most common, appearing in 58 cases, is that agents check their own work too leniently. In 34 cases, agents spot a problem but document it instead of fixing it. In 21 cases, they unilaterally shrink the scope of the task.

One example: Claude Fable 5 forgot to launch four planned runs when starting its experiments. Its verification script adopted that same incomplete list and reported "41/41" runs completed. The agent trusted that result until the very end. In a different task, GPT-5.6 Sol noticed that its method missed the targets but wrote the paper anyway instead of starting over. DeepSeek V4 Pro dropped experiments because they were supposedly too slow for the "remaining time budget," even though there was no time limit.

Two work logs side by side. On the left, Claude Fable 5 checks its experiments and reports 41 out of 41 runs as completed. On the right, GPT-5.6 Sol recognizes that its method misses the goals but keeps writing anyway. Both passages are highlighted in red as errors. — Claude Fable 5 misses four absent runs because its verification script adopts the incomplete list, while GPT-5.6 Sol writes the paper despite missing its targets. | Image: Kim et al

There are counterexamples, too: in a quantum task, GPT-5.6 Sol caught a fundamental error, stopped the running experiments, and recomputed everything from scratch.

More compute time doesn't help

Longer runtimes didn't lead to better results. DeepSeek V4 Pro and GLM-5.3 ran the longest at median times of 19.5 and 13.9 hours but scored poorly. GPT-6 Astra got by with a median of just 1.3 hours and delivered the best results.

Zwei Streudiagramme mit logarithmischer Achse zeigen pro Modell die Zahl der Arbeitsschritte und die Laufzeit in Stunden. DeepSeek V4 Pro braucht im Median 19,5 Stunden, GPT-6 Astra 1,3 Stunden. — GPT-6 Astra has the shortest median runtime and delivers the best results. | Image: Kim et al.

Because the strongest model also served as the evaluator, the researchers ran a cross-check: the original papers' authors judged anonymized AI papers themselves. On whether a paper matched the given research direction, the automated system agreed with the experts in every case.

Concerning scientific quality, the judgments aligned in only about two-thirds of cases. One expert criticized, for instance, that the papers didn't tell a story. Judgments like these remain the job of human reviewers, the authors say.

The study's code and data are publicly available, and the project page hosts a leaderboard of the tested models.

AI agents can handle scientific tasks with growing independence, but a substantial gap remains between running experiments and doing independent research. A study from August already showed how much AI agents struggle with unpublished research questions. Epoch AI and Anthropic recently found that agents tend to present their results too favorably and don't sufficiently question failed approaches.

The picture looks much better for well-defined tasks: GPT-6 Astra solved 49 out of 50 unpublished problems in theoretical physics, though those solutions could be verified unambiguously. An analysis of more than 700 tasks from AI development also showed that while agents take on more and more practical work, humans still make the key research decisions.