arXiv cs.AI· Bin Shao, Jixiang Li, Xinyue Zhang, Baishun Yang, Zhiyang Liu, Weichao Wang·· 12 小时前AI 评分27
INCARBench：评估大语言模型配置 VASP INCAR 的基准
INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models
AI 导读
研究者提出 INCARBench，用于评测大语言模型在 VASP 输入配置上的生成与修复能力。对 19 个模型配置的评估显示，前沿模型间能力差异显著，语义与策略准确率虽高，但任务关键正确率明显偏低。错误集中在 DFT+U、磁性与关联材料等物理耦合设置，修复评测还表明纠正错误配置与保留已有效配置是两种不同能力。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly being integrated into first-principles computational workflows, yet their ability to configure scientific calculations remains poorly understood. Here, we introduce INCARBench, a benchmark for evaluating LLMs on input configuration for the Vienna Ab initio Simulation Package (VASP) through both configuration generation and repair tasks. Evaluating 19 model configurations reveals substantial capability differences among current frontier models. While several models achieve high semantic and policy accuracy, task-critical correctness remains substantially lower, demonstrating that parameter-level correctness does not necessarily imply scientifically valid configurations. Failure analysis shows that errors concentrate in physically coupled settings involving DFT+$U$, magnetism, and correlated materials, where multiple constraints must be satisfied simultaneously. Repair evaluation further reveals that correcting incorrect settings and preserving already-valid configurations are distinct capabilities, with configuration preservation remaining a major challenge. These findings establish scientific configuration as a measurable capability of large language models and provide a foundation for developing more reliable AI systems for computational materials science.
| Subjects: | Materials Science (cond-mat.mtrl-sci); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.23571 [cond-mat.mtrl-sci] |
| (or arXiv:2606.23571v2 [cond-mat.mtrl-sci] for this version) | |
| https://doi.org/10.48550/arXiv.2606.23571 arXiv-issued DOI via DataCite |
Submission history
From: Bin Shao [view email]
[v1]
Mon, 22 Jun 2026 16:38:42 UTC (2,563 KB)
[v2]
Tue, 23 Jun 2026 04:48:23 UTC (2,563 KB)
来源：arXiv cs.AI · arxiv.org