ORACLE-SWE：量化关键信息信号对 SWE 智能体的贡献

Authors:Kenan Li, Qirui Jin, Liao Zhu, Xiaosong Huang, Yijia Wu, Yikai Zhang, Xin Zhang, Zijian Jin, Yufan Huang, Elsie Nallipogu, Chaoyun Zhang, Yu Kang, Saravan Rajmohan, Qingwei Lin, Wenke Lee, Dongmei Zhang

View PDF HTML (experimental)

Abstract:Recent advances in Language Model (LM) agents have increasingly focused on repository-level software engineering (SWE) tasks, leading to various agentic workflows, training methods and failure analysis. Interestingly, rather than directly generating or evaluating edit patches, these approaches first examine and improve intermediate information signals collected during agent exploration, including Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage. Although different studies examine different subsets of these signals, no work has systematically compared their relative effectiveness in improving final task success or measured the remaining space for improvement in each signal. To address this gap, this work quantifies the contributions of both oracle and agent-extracted signals to final task success and measures the gap between current agent-found signals and their oracle counterparts. We hope our findings could suggest future research emphasis in SWE agents. We release our source code and oracle-signal dataset as a reusable methodology to benefit future signal analysis of SWE agents.

Comments:	Under peer review; 31 pages, 9 figures, 11 tables
Subjects:	Multiagent Systems (cs.MA); Computation and Language (cs.CL); Software Engineering (cs.SE)
ACM classes:	I.2.7; I.2.5
Cite as:	arXiv:2604.07789 [cs.MA]
	(or arXiv:2604.07789v3 [cs.MA] for this version)
	https://doi.org/10.48550/arXiv.2604.07789 arXiv-issued DOI via DataCite

Submission history

From: Kenan Li [view email]
[v1] Thu, 9 Apr 2026 04:37:24 UTC (639 KB)
[v2] Thu, 28 May 2026 09:34:24 UTC (620 KB)
[v3] Thu, 15 Oct 2026 14:43:29 UTC (527 KB)