arXiv cs.CL· Lucas H. McCabe, Sarah Lukens, H. Howie Huang·· 3 天前AI 评分25
LLM 智能体能否高效识别家电故障？来自模拟家电诊断的洞察
Can LLM Agents Efficiently Identify Appliance Faults? Insights From Diagnosing a Simulated Appliance
AI 导读
研究者提出 CoffeeBench，一个用模拟家电测试 LLM 智能体故障诊断能力的任务。被测 LLM 难以仅凭传感器信息稳定识别故障，但表现最好的模型能迭代缩小故障选项空间并最终完成诊断。分析还发现，初始诊断不确定性和 LLM 评判的推理合理性是预测诊断成功的高信号指标。
正文 · 原文
Abstract:In so-called "Industry 4.0," integrated sensors, analytics tools, and other digital technologies supplement industrial workflows, potentiating enhanced visibility and automation. Prognostics and health management (PHM) and natural language processing (NLP) practitioners alike have explored the application of large language models (LLMs) to industrial asset management, but many prior works' focus on exam-like benchmarks has limited their assessment of LLMs' ability to automate practical operations. Herein, we introduce CoffeeBench, a fault diagnostics task for LLM agents, using a simulated appliance. The LLMs we evaluate struggle to consistently identify faults from the sensor information alone, but the best-performing ones can iteratively refine the fault-option space to eventually obtain a successful diagnosis. Our subsequent analyses, therefore, focus on: differentiating models by diagnostic efficiency (RQ1); examining the lagging performance of the three worst models (RQ2); and identifying viable heuristics for selective prediction (RQ3), including that initial diagnostic uncertainty and LLM-judged soundness of reasoning are high-signal predictors of diagnostic success. In addition to our benchmark results (Section 3.1), we provide practical insights for improving the performance of the open-weight models on CoffeeBench, such as by explicitly tracking episode state on their behalf to reduce timeouts (Section 3.2) or applying a deliberation loop to refine diagnostic hypotheses (Section 3.3).
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19291 [cs.CL] |
| (or arXiv:2610.19291v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19291 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lucas McCabe [view email]
[v1]
Tue, 13 Oct 2026 23:12:32 UTC (337 KB)
来源：arXiv cs.CL · arxiv.org