arXiv cs.AI· Devin Yasith De Silva, Dhaval Patel, Christodoulos Constantinides, Shuxin Lin, Nianjun Zhou, Dharmashankar Subramanian, Paul J Adams, Sal Rosato, Nicolas Constantinides, Deborah L. McGuinness, Jayant Kalagnanam·· 14 小时前AI 评分31
DiagnosticIQ：基于符号规则的 LLM 工业维护动作推荐基准
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
AI 导读
研究者发布 DiagnosticIQ 基准，含 6690 道多选题，由 16 类资产的 118 组专家规则-动作对生成，用于评测 LLM 将符号规则转化为维护动作的能力。基准测试 29 个 LLM 与 4 个嵌入基线，前三名模型 Macro 分差在 1 分以内、规则级检验下不可区分。但在干扰项扩展下领先模型掉 13.8 Macro 分，条件反转时仍选原答案的比例达 63%。
正文 · 原文
Authors:Devin Yasith De Silva, Dhaval Patel, Christodoulos Constantinides, Shuxin Lin, Nianjun Zhou, Dharmashankar Subramanian, Paul J Adams, Sal Rosato, Nicolas Constantinides, Deborah L. McGuinness, Jayant Kalagnanam
Abstract:Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger on sensor conditions and prompt technicians to perform corrective actions. The bottleneck is not detection but response: translating rules into maintenance steps requires asset-specific knowledge gained through years of practice. We investigate whether LLMs can serve as decision support for this rule-to-action step and introduce \ours{}, a benchmark of 6{,}690 multiple-choice questions generated from 118 expert-authored rule-action pairs across 16 asset types, with experts reviewing the generated prompts for layout and conversational flow. We contribute (i) a symbolic-to-MCQA pipeline normalizing rules to Disjunctive Normal Form with embedding-based distractor sampling, (ii) five variants probing distinct failure modes (Pro, Pert, Verbose, Aug, Rationale), and (iii) a benchmark of 29 LLMs and 4 embedding baselines. A human evaluation (9 practitioners, mean 45.0\%) confirms \ours{} requires specialist knowledge beyond operational experience. Three findings stand out: the frontier has closed, with the top three LLMs within one Macro point and inseparable under rule-level tests. Yet \ours{}\,Pro exposes brittleness, with the leader losing 13.8 Macro points and smaller models losing over 60\% of their accuracy under distractor expansion. \ours{}\,Aug exposes pattern-matching: under condition inversion, the leading model still selects the original answer 63\% of the time. Frontier models handle template-style fault detection but break under structural perturbation.
| Comments: | 48 pages, 27 tables, 24 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.08614 [cs.AI] |
| (or arXiv:2605.08614v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08614 arXiv-issued DOI via DataCite |
Submission history
From: Dhaval Patel Dr [view email]
[v1]
Sat, 9 May 2026 02:17:39 UTC (6,633 KB)
[v2]
Sat, 24 Oct 2026 19:43:24 UTC (3,908 KB)
来源：arXiv cs.AI · arxiv.org