arXiv cs.AI· Yuyang Wu, Yue Huang, Shuaike Shen, Xujian Wang, Shuhao Zhang, Qiyao Xue, Weichen Liu, Runtian Gao, Jian Ma, Xiangliang Zhang, Olexandr Isayev·· 6 天前AI 评分26
ChemCost：评估 LLM 智能体的化学反应成本推理能力
Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning
AI 导读
研究者提出 ChemCost 基准，用 1,427 个可评估反应检验 LLM 智能体能否完成化学采购成本估算，覆盖 2,261 种化学品和 230,775 条供应商报价。最强智能体在干净输入下仅有 50.6% 的结果落在 25% 相对误差内，注入别名、数量表达等噪声后性能大幅下降。阶段级分析显示失败主要源于解析脆弱、证据整合低效、包装选择无效和工具使用不收敛。
正文 · 原文
Authors:Yuyang Wu, Yue Huang, Shuaike Shen, Xujian Wang, Shuhao Zhang, Qiyao Xue, Weichen Liu, Runtian Gao, Jian Ma, Xiangliang Zhang, Olexandr Isayev
Abstract:Large Language Models (LLMs) have become increasingly capable as tool-using agents, with benchmarks spanning diverse general agentic tasks. Yet rigorous evaluation of scientific tool use remains limited. In chemistry, recent agents can plan syntheses and invoke domain-specific tools, but evaluations often rely on curated demonstrations, expert assessment, or LLM-as-judge scoring rather than exact, judge-free ground truth. We address this gap with chemical procurement cost estimation, a practical task in which an agent must ground chemical identities, retrieve supplier quotes, select valid purchasable packs, normalize quantities, and compute cost from a reaction description. We introduce ChemCost, a benchmark of 1,427 evaluable reactions grounded to a frozen pricing snapshot covering 2,261 chemicals and 230,775 supplier quotes, supporting scalar scoring and stage-level diagnosis of grounding, retrieval, procurement, and arithmetic failures. To evaluate robustness, we further construct controlled noise-injected views that perturb chemical aliases, quantity expressions, missing fields, and input formatting. Experiments with frontier, open-weight, and chemistry-specialized LLM agents show that tool access is necessary but insufficient for solving the task. The strongest agents reach only 50.6% accuracy within 25% relative error on clean inputs and degrade substantially with realistic noise. Stage-level analysis further shows that failures arise from brittle parsing, ineffective evidence integration, invalid pack selection, and non-convergent tool use.
| Comments: | 9 pages, 5 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.07251 [cs.AI] |
| (or arXiv:2605.07251v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07251 arXiv-issued DOI via DataCite |
Submission history
From: Yuyang Wu [view email]
[v1]
Fri, 8 May 2026 05:19:59 UTC (2,903 KB)
[v2]
Fri, 9 Oct 2026 04:18:27 UTC (2,900 KB)
来源：arXiv cs.AI · arxiv.org