跳到正文
arXiv cs.MA· John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu·· 9 小时前AI 评分40

BuggyMLE:用注入 bug 扩展机器学习工程任务难度

BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection

AI 导读

研究者提出 Code Complexity Index(CCI)难度指标,并基于它构建 BuggyMLE 框架,向可运行的 ML 仓库注入受控变异,生成超 6,600 个 bug 注入仓库变体和近 260K 行注入代码。

正文 · 原文

Authors:John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu

View PDF HTML (experimental)

Abstract:Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks ($R^2 = 0.951$), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA); Software Engineering (cs.SE)
Cite as: arXiv:2611.25088 [cs.AI]
  (or arXiv:2611.25088v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.25088

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: John Wu [view email]
[v1] Wed, 18 Nov 2026 19:26:15 UTC (703 KB)

来源:arXiv cs.MA · arxiv.org