arXiv cs.AI· Yanis Merzouki, Coral Izquierdo, Matei Ignuta-Ciuncanu, Marcos Gomez-Bracamonte, Riccardo Maggioni, Alessandro Lombardi, Camilla Mazzoleni, Federico Martelli, Balazs Gunther, Jonas Petersen, Philipp Petersen·· 6 天前AI 评分31
FactoryBench：评估工业机器理解能力的基准
FactoryBench: Evaluating Industrial Machine Understanding
AI 导读
研究者推出 FactoryBench，用于评估时间序列模型和 LLM 在工业机器人遥测数据上的机器理解能力。该基准包含超 7 万条 Q&A，覆盖因果阶梯四个层级和五种答案格式，数据源自 UR3 协作机器人与 KUKA KR10 工业机械臂采集的 FactoryWave 数据集。六款前沿 LLM 零样本评测均未在结构化层级超过 50%，决策任务不超过 18%。
正文 · 原文
Authors:Yanis Merzouki, Coral Izquierdo, Matei Ignuta-Ciuncanu, Marcos Gomez-Bracamonte, Riccardo Maggioni, Alessandro Lombardi, Camilla Mazzoleni, Federico Martelli, Balazs Gunther, Jonas Petersen, Philipp Petersen
Abstract:We introduce FactoryBench, a benchmark for evaluating time-series models and LLMs on machine understanding over industrial robotic telemetry. Q&A pairs are organized along four causal levels (state, intervention, counterfactual, decision) instantiating Pearl's ladder of causation, and span five answer formats: four structured formats are scored deterministically and free-form answers are scored by an LLM-as-judge voting protocol. We propose a scalable Q&A generation framework built around structured question templates, present FactoryWave (a dense, multitask, multivariate sensor dataset collected from a UR3 cobot and a KUKA KR10 industrial arm), and construct FactoryBench as a large-scale benchmark of over 70k Q&A items grounded in roughly 15k normalized episodes from FactoryWave, AURSAD, and voraus-AD. Zero-shot evaluation of six frontier LLMs shows that no model exceeds 50% on structured levels or 18% on decision-making, revealing a wide gap between current models and operational machine understanding.
| Comments: | Oral at PUDM, NeurIPS 2026. Code: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| MSC classes: | cs.AI (primary), cs.RO, cs.LG, cs.CL |
| ACM classes: | I.2.6; I.2.9 |
| Cite as: | arXiv:2605.07675 [cs.AI] |
| (or arXiv:2605.07675v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07675 arXiv-issued DOI via DataCite |
Submission history
From: Jonas Petersen [view email]
[v1]
Fri, 8 May 2026 12:47:05 UTC (13,609 KB)
[v2]
Thu, 8 Oct 2026 21:27:23 UTC (14,138 KB)
来源：arXiv cs.AI · arxiv.org