arXiv cs.AI· Xiaobo Chen, Genliang Zhu, Ziyuan Wang, Chu Wang·· 12 小时前AI 评分52
研究：API 迁移后 LLM 代码修复存在静默错误，常规测试放行 41% 错误修复
Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations
AI 导读
一项预注册研究用 73 个任务覆盖 CPython、某 URL 库和 SQLite 的 60 项有记录变更，在真实运行时上执行候选修复后发现，基于典型输入的断言门禁放行了 0.410 的错误修复。
正文 · 原文
Abstract:After a runtime or API migration, an LLM-repaired procedure can run cleanly yet return wrong results, and a release check on typical test inputs ships it. We report a preregistered study of 73 tasks covering 60 documented changes in CPython, this http URL and SQLite. Every candidate is executed on the real runtimes against withheld contracts. Over the complete repair space, the all-assertions gate on typical inputs released 0.410 of wrong repairs. One edge input written from the release note cut this to 0.065, and to 0.061 on 15 unseen clusters. Three inputs from a blind GPT-5.6-luna writer that never saw the program reached 0.070. Two further frozen stages enlarged the menu to 1491 fragments and then added 233 hard distractors that match the legacy output on typical inputs. The edge input (0.058) and the frontier-model writer (0.058) passed their preregistered tests. Yet 0.168 and 0.153 of the hard wrong programs still passed. Two local-model writers failed their preregistered tests. Across four model families, the edge input cut wrong releases per attempt in all three open-weight families on all three menus, for example from 0.233 to 0.057, without lowering correct completion. Self-tests helped less. A blind AI audit flagged 34 contracts as stricter than their requirements. Dropping the disputed checks changed 0 verdicts, but class-label disputes remain open. No model-written code was executed. Artifacts are available from the authors on request.
| Comments: | 30 pages, 2 figures. Preregistered; artifacts are available from the authors on request |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| ACM classes: | D.2.5; D.2.7; I.2.2 |
| Cite as: | arXiv:2611.08269 [cs.SE] |
| (or arXiv:2611.08269v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08269 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaobo Chen [view email]
[v1]
Thu, 5 Nov 2026 16:41:32 UTC (105 KB)
来源：arXiv cs.AI · arxiv.org