如何为知识工作设计 AI 智能体基准测试

View PDF HTML (experimental)

Abstract:Agent benchmarks have attracted substantial attention; yet, benchmarks for AI agents performing knowledge work are still mostly described only by their tasks, environments, and metrics. Starting from the definition of knowledge work, this paper derives four other design choices that often remain implicit. Three research threads on how knowledge work is organized each yield one choice: studies of professions, situated action, and boundary objects and coordination, while validity theory provides the fourth. Following this reasoning, we propose a benchmark representation centered on the work being performed. Based on this representation, we propose reporting practices for four largely missing fields: represented activity, tested setting, required work product, and evaluated result. Benchmarks designed in disparate ways thereby share a common description and can be compared side by side. To support the reporting of work activities, we cluster O*NET task statements into 18 shared work activities through unsupervised learning paired with human review. We then use this representation to analyze cases from three popular knowledge work benchmarks: GDPval, OfficeQA Pro, and APEX-SWE. Across all three benchmarks, the scope covered by the evaluated result is narrower than that of the required work product. We audited the tests of all 25 public APEX-SWE Integration tasks, independently coded by two authors. We found from the test code that in 13 of these tasks, a deliverable can alter established content from the source data and still pass all tests. We synthesize these four fields into a one-page record, accompanied by a worked example and a fillable template.

Submission history

From: Yining Hua [view email]
[v1] Fri, 22 May 2026 06:03:01 UTC (1,209 KB)
[v2] Mon, 24 Aug 2026 13:42:48 UTC (2,081 KB)
[v3] Tue, 3 Nov 2026 23:25:29 UTC (991 KB)