arXiv cs.CL· Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth·· 12 小时前AI 评分34
AuditKIT：面向大语言模型与智能体系统评测、评判与比较的统一评估框架
AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems
AI 导读
AuditKIT 是一个源码可获取的评估库，将基准准确率、评判模型评估、运行时性能、工具调用行为和检索质量记录在同一结果表示中，并支持方向感知的指标契约与配置指纹。针对智能体，它提供按轮次的工具调用匹配、区分并行与串行执行，并将缺失或不可观测的证据标记为未计分而非失败。论文通过覆盖模型比较、服务运行时、检索、工具调用和互操作性的七个案例，展示答案抽取、运行时行为和指标选择如何改变评估结果。
正文 · 原文
Abstract:Evaluating language models and agents requires combining benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality. Differences in interfaces, metric semantics, and trace formats complicate these comparisons and can hide failures in agentic workflows. We present AuditKIT, a source-available evaluation library that records these measurements in one result representation, with direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends. For agents, it provides turn-aware tool-call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory-level diagnostics. Missing or unobservable evidence is marked unscored rather than counted as failure. For retrieval-augmented generation, it combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability show how answer extraction, runtime behavior, and metric selection change evaluation outcomes.
| Comments: | 25 pages, 2 figures, 13 tables. Code: this https URL. Shreeyans Arora and Utsav Avaiya contributed equally; work done while at Lexsi Labs |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.22511 [cs.CL] |
| (or arXiv:2610.22511v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22511 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hem Gosalia [view email]
[v1]
Thu, 15 Oct 2026 18:05:10 UTC (241 KB)
来源：arXiv cs.CL · arxiv.org