跳到正文
arXiv cs.AI· Hanfeng Liu, Zhenyu Luo, Dongchao Zhang, Zexi Liu, Zulong Chen, Jinxin Hu, Jia Xu, Zeyi Wen·· 4 小时前AI 评分35

OfficePrism:评测 AI 智能体在企业各业务职能单域任务上的表现

OfficePrism: Benchmarking Agents on Single-Domain Operations Across Enterprise Business Functions

AI 导读

OfficePrism 基准用 420 个单域企业任务评测 AI 智能体,覆盖七个业务职能并设 L1–L3 任务结构层级。在 124 组模型—harness—工具包配置中,自然语言与数据库联合通过率为 22.6%–76.4%,中位数 67.9%;全部配置 L1 得分最高,122 组中 L2 高于 L3。

正文 · 原文

View PDF HTML (experimental)

Abstract:Broad workplace scores can mix competence within one business function with the extra cost of coordinating across functions and systems, making failures hard to localize. OfficePrism evaluates this prerequisite through 420 single-domain enterprise tasks spanning seven business functions under shared, benchmark-defined L1--L3 task-structure levels. Each task runs in a fresh environment combining a shared synthetic organization with one function-specific domain and yped state access. Our pipeline uses grounded records, business rules, and tool schemas to jointly construct a Chinese-language ticket, a hidden executable reference witness, and an expected response, followed by automated filtering and human curation. The same task set is evaluated through full-schema and search-and-dispatch packages. Across 124 model--harness--package configurations, the per-case joint natural-language-and-database pass rate ranges from 22.6% to 76.4%, with a median of 67.9%. L1 is highest in all 124 configurations, and L2 exceeds L3 in 122. Among 50 configurations sharing the same harness and model labels across both packages, search scores higher in 30, ties in one, and scores lower in 19.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
Cite as: arXiv:2611.22262 [cs.CL]
  (or arXiv:2611.22262v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.22262

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zhenyu Luo [view email]
[v1] Tue, 17 Nov 2026 06:10:32 UTC (3,643 KB)

来源:arXiv cs.AI · arxiv.org