WorkflowPerturb：面向多智能体工作流指标评估的校准压力测试基准

View PDF HTML (experimental)

Abstract:Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise orchestration. Operating them exposes a recurring change-management problem. Routine updates, such as re-running an input, swapping the LLM, or refactoring an agent's prompt or orchestration code, often produce workflows that differ substantially from validated references. Engineers then lack a principled way to decide whether a change is safe to ship. Automatic workflow evaluation is the natural tool, but in practice metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation. We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. It contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities of 10%, 30%, and 50%. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates. Our results characterize systematic differences across families and support severity-aware interpretation of workflow evaluation scores in change-management settings. WorkflowPerturb is publicly available at this https URL .

Comments:	Accepted to EMNLP 2026 Industry Track. 18 pages
Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2602.17990 [cs.AI]
	(or arXiv:2602.17990v4 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2602.17990 arXiv-issued DOI via DataCite

Submission history

From: Sharad Agarwal [view email]
[v1] Fri, 20 Feb 2026 04:54:31 UTC (1,060 KB)
[v2] Sun, 14 Jun 2026 22:14:32 UTC (399 KB)
[v3] Thu, 10 Sep 2026 20:04:43 UTC (404 KB)
[v4] Thu, 15 Oct 2026 17:50:57 UTC (404 KB)