跳到正文
arXiv cs.AI· Priyam Sahoo, Gaurav Mittal, Xiaomin Li, Shengjie Ma, Benjamin Steenhoek, Pingping Lin, Yu Hu·· 4 小时前AI 评分58

AgentLens 揭示 SWE-Agent 评测中的 Lucky Pass 问题

AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

AI 导读

论文指出仅看最终补丁是否通过测试的二元评测会把侥幸通过和严谨解法等同,在 60 个 SWE-bench Verified 任务、8 个模型后端的 2,614 条 OpenHands 轨迹中,1,815 条评测子集里有 10.7% 的通过轨迹属于 Lucky Pass。

正文 · 原文

View PDF HTML (experimental)

Abstract:Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks. Of these, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and release AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens builds PTA references by merging multiple passing solutions for the same task, and uses a context-sensitive intent labeler to assign actions to Exploration, Implementation, Verification, or Orchestration based on trajectory history rather than tool identity alone. On AgentLens-Bench, the quality score separates passing trajectories into Lucky, Solid, and Ideal tiers and further decomposes Lucky Passes into five recurring mechanisms. Across the eight model backends, Lucky rates range from 0.5% to 23.2%, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We publicly release AgentLens-Bench and the AgentLens SDK at this https URL, with a project page at this https URL.
Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
Cite as: arXiv:2605.12925 [cs.SE]
  (or arXiv:2605.12925v5 [cs.SE] for this version)
  https://doi.org/10.48550/arXiv.2605.12925

arXiv-issued DOI via DataCite

Submission history

From: Priyam Sahoo [view email]
[v1] Wed, 13 May 2026 03:00:57 UTC (545 KB)
[v2] Wed, 27 May 2026 20:12:24 UTC (542 KB)
[v3] Tue, 2 Jun 2026 15:49:40 UTC (542 KB)
[v4] Mon, 5 Oct 2026 02:54:23 UTC (549 KB)
[v5] Tue, 17 Nov 2026 07:41:17 UTC (3,492 KB)

来源:arXiv cs.AI · arxiv.org