跳到正文
arXiv cs.AI· Gengyue Dong, Ye Wei, Zheng Zhu, Yidong Han, Jiale Miao, Xiaojun Jia, Guanghui Li·· 4 小时前AI 评分38

SkillBuffBench:衡量 Agent Skills 在不同 Harness、模型、技能访问模式与记忆下的贡献

SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory

AI 导读

研究者提出 SkillBuffBench,一个将 Skill 访问分为无 Skill、精选 Skill、从已安装库自动调用三档的基准与配对评估协议,并与记忆开关、四种 agent harness 及多个模型交叉组合。

正文 · 原文

View PDF HTML (experimental)

Abstract:Agent Skills, file-based packages of procedural knowledge that a coding agent can discover and load on demand, are now shipped by every major agent harness, yet the evidence for how much they help comes almost entirely from settings in which the right Skill is hand-picked for the task and the agent starts from a blank slate. Neither holds in deployment, where an agent chooses from whatever library happens to be installed and frequently carries memory from earlier work. We present SkillBuffBench, a benchmark and paired evaluation protocol that treats Skill access as a three-level factor (no Skill, a curated Skill, and automatic invocation from an installed library), crosses it with memory (off and on), with four agent harnesses and with multiple models (the state-of-the-art proprietary models from Anthropic, Google and OpenAI, and the state-of-the-art open-weight models from Chinese providers), every model being run under every harness, and evaluates the result on research tasks contributed by domain experts together with the Skills they use, currently in materials science and biology, with the suite extended on a rolling basis. Every run is marked by a deterministic verifier; uncertainty is reported with a paired, task-level bootstrap; and cost, agent rounds and wall-clock time are reported alongside the score rather than folded into it. This version of the paper describes the design of the benchmark, its evaluation protocol and its current tasks and Skills. The evaluation is in progress: results will be added once the runs are complete, and the task suite, results and analysis will continue to be updated on a rolling basis.
Comments: 18 pages, 2 figures, 6 tables. Work in progress: benchmark design and protocol; results to be added in a later version
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2611.22017 [cs.AI]
  (or arXiv:2611.22017v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22017

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Gengyue Dong [view email]
[v1] Tue, 17 Nov 2026 00:12:40 UTC (27 KB)

来源:arXiv cs.AI · arxiv.org