arXiv cs.AI· Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao·· 10 小时前AI 评分39
企业分析智能体的评估框架对比研究：稳定性、检测与发现
Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
AI 导读
一项三阶段对比研究比较了基于执行轨迹的 LLM 评判、纯文本 LLM 评判加红队测试、确定性启发式三种评估框架，在冻结执行轨迹上评测两个分析智能体。97% 的配置达到稳定性阈值，基于轨迹的框架在 72 个真实测试用例上取得最佳精确率-召回率平衡（F1=0.90），纯文本评估召回率相当但误报多 5 倍。
正文 · 原文
Abstract:Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
| Comments: | Accepted to KDD 2026, Jeju, Korea. 9 pages, 3 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.14493 [cs.AI] |
| (or arXiv:2611.14493v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14493 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Howard Zhang [view email]
[v1]
Tue, 10 Nov 2026 18:50:06 UTC (283 KB)
来源：arXiv cs.AI · arxiv.org