arXiv cs.AI· Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, Qinghua Ni, Jun Huang, Qiang Zhang, Yue Zhou, Zhihang Zhong, Yonglin Tian, Fei-Yue Wang, Xue Yang·· 12 小时前AI 评分38
BEAKER：面向自动驾驶化学实验室具身大脑的专家精选基准
BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories
AI 导读
研究团队推出 BEAKER，一个面向自驱动化学实验室（SDCL）具身大脑的评测基准，含 1000 条由化学与具身 AI 专家人工编写并审核的问答样本，覆盖 11 种图像与视频任务类型。
正文 · 原文
Authors:Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, Qinghua Ni, Jun Huang, Qiang Zhang, Yue Zhou, Zhihang Zhong, Yonglin Tian, Fei-Yue Wang, Xue Yang
Abstract:Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.
| Comments: | Accepted by NeurIPS 2026 (Track on Evaluations and Datasets, Spotlight) |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.06911 [cs.AI] |
| (or arXiv:2611.06911v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06911 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fei Lin [view email]
[v1]
Wed, 4 Nov 2026 18:57:17 UTC (20,405 KB)
来源：arXiv cs.AI · arxiv.org