arXiv cs.AI· Shan Peng, Shuaiqi Cheng, Miao Peng, Chenyi Zi, Jia Li·· 12 小时前AI 评分42
DMAIC-Bench：面向多阶段分析任务中智能体的自我反思与自我约束评测基准
DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks
AI 导读
DMAIC-Bench 用 97 个种子任务、覆盖 4 个行业 15 个工业案例，评测 LLM 智能体在多阶段分析中能否依据中间证据决定继续、改道或停止。13 个来自 6 家提供商的模型在匹配 GO 决策上的执行级路由准确率平均为 92%，但在 NO-GO 决策上仅 19%。
正文 · 原文
Abstract:Multi-stage analytical tasks require LLM agents to use intermediate evidence to decide whether to continue, redirect or stop. Self-reflection, in which an agent critiques its output, is widely studied, yet a critique does not show whether evidence changed the next action. We call that capability self-restraint. Existing agent benchmarks largely measure task completion or output quality and leave it untested. DMAIC-Bench evaluates it with 97 seeded tasks across 15 industrial cases in four sectors, built around Lean Six Sigma's explicit statistical prerequisites. Paired counterfactual variants require opposite actions at the same workflow position, distinguishing warranted restraint from indiscriminate stopping. We assess execution traces with a private scorer and final claims through independent expert review. Across 13 models from six providers, execution-level routing accuracy averages 92% on matched GO decisions and 19% on NO-GO decisions, counting any forbidden downstream computation as a route error. Execution traces show agents reporting correct diagnostics before performing analyses that those results disallow. Separately, blinded expert review of the 11 audited agents estimates that reports act on invalid evidence at 47% of reached NO-GO decisions, with an interval from 31% to 61%. Explicit stopping permission helps some models without consistently restoring correct decisions at both GO and NO-GO forks. Sixteen life-sciences tasks in bulk ribonucleic acid sequencing (RNA-seq) and case-control genome-wide association studies (GWAS) show that the same framework can evaluate evidence-dependent decisions beyond industrial workflows. Project code and data are available at: this https URL.
| Comments: | 54 pages, 14 figures. Code and data: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.32305 [cs.AI] |
| (or arXiv:2610.32305v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32305 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shan Peng [view email]
[v1]
Fri, 23 Oct 2026 08:04:10 UTC (1,397 KB)
来源：arXiv cs.AI · arxiv.org