arXiv cs.AI· Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu·· 13 小时前AI 评分58
SWE-Skills-Bench：Agent 技能在真实软件工程中真的有用吗
SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
AI 导读
研究者提出 swe-skills-bench，一个以技能为中心、需求驱动的基准，用于衡量单个 Agent 技能在真实软件工程任务上的提升幅度。该基准从 98 个公开 SWE 技能出发，每个技能配对一个固定 commit 的真实 GitHub 仓库，在四种 Agent 模型配置下分别做无技能与有技能的配对评测，共 3920 次。
正文 · 原文
Abstract:Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear, and no standard procedure tests an individual skill before installation. We present swe-skills-bench, a skill-centric, requirement-driven benchmark that measures how much individual agent skills improve performance on real-world software engineering (SWE) tasks. We start from 98 public SWE skills, each paired with a real GitHub repository pinned at a fixed commit. For each skill, we derive repository-specific requirements, create executable verifiers, and build a reproducible environment. Under each of four agent-model configurations, we run every task once without the skill and once with it, yielding 3,920 paired evaluations. For each requirement, we compute a paired delta, i.e., the change in outcome when the skill is added. We then aggregate these deltas across a skill's ten requirements into a per-skill utility profile, and use a statistical test to classify each skill as Golden (significantly helpful), Ineffective (no significant effect), or Harmful (significantly detrimental). Our results show that skills provide limited benefits: 92 of 98 skills have no statistically significant effect, and adding skills raises the average pass ratio only from 76.6% to 76.9%. The few helpful skills supply procedures that the agent would otherwise omit (gains of up to $+17.7$ points). In contrast, harmful skills anchor the agent on their own templates and override the task's explicit instructions (losses of up to -17.0 points). Our code is available at this https URL.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.15401 [cs.SE] |
| (or arXiv:2603.15401v2 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2603.15401 arXiv-issued DOI via DataCite |
Submission history
From: Tingxu Han [view email]
[v1]
Mon, 16 Mar 2026 15:16:31 UTC (1,343 KB)
[v2]
Tue, 27 Oct 2026 09:54:21 UTC (1,279 KB)
来源：arXiv cs.AI · arxiv.org