AblationBench：评估 AI 实证研究中的消融实验自动规划能力

View PDF HTML (experimental)

Abstract:Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge. A key mechanism to obtain such insights is through ablation experiments. To this end, we introduce AblationBench, a benchmark suite for evaluating agents on ablation planning tasks in empirical AI research. It includes two tasks: AuthorAblation, which helps authors propose ablation experiments based on a method section and contains 83 instances, and ReviewerAblation, which helps reviewers find missing ablations in a full paper and contains 350 instances. For both tasks, we develop LM-based judges that serve as an automatic evaluation framework. Our experiments with frontier LMs show that these tasks remain challenging, with the best-performing LM system identifying only 45% of the original ablations on average, below human-level performance. We observe an inverse performance trend between the author and reviewer tasks, which we attribute to differences in model grounding. Lastly, we analyze the limitations of current LMs on these tasks, and find that chain-of-thought prompting outperforms an agent-based approach. Our data is available on this https URL, and our code is available on this https URL .

Comments:	Findings of the EMNLP 2026; Project page: this https URL
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2507.08038 [cs.CL]
	(or arXiv:2507.08038v4 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2507.08038 arXiv-issued DOI via DataCite

Submission history

From: Talor Abramovich [view email]
[v1] Wed, 9 Jul 2025 12:07:38 UTC (1,760 KB)
[v2] Sat, 31 Jan 2026 13:22:50 UTC (2,299 KB)
[v3] Mon, 1 Jun 2026 08:32:26 UTC (2,489 KB)
[v4] Fri, 6 Nov 2026 15:07:38 UTC (2,487 KB)