arXiv cs.MA· John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu·· 9 小时前AI 评分40
BuggyMLE:用注入 bug 扩展机器学习工程任务难度
BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
AI 导读
研究者提出 Code Complexity Index(CCI)难度指标,并基于它构建 BuggyMLE 框架,向可运行的 ML 仓库注入受控变异,生成超 6,600 个 bug 注入仓库变体和近 260K 行注入代码。
正文 · 原文
Authors:John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu
Abstract:Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks ($R^2 = 0.951$), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.25088 [cs.AI] |
| (or arXiv:2611.25088v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.25088 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: John Wu [view email]
[v1]
Wed, 18 Nov 2026 19:26:15 UTC (703 KB)
来源:arXiv cs.MA · arxiv.org