arXiv cs.CL· Leilei Ding, Shumin Wang, Yuting Huang, Fanqi Wan, Yinmin Zhang, Qi Han, Yiming Xu, Feiyuan Zhang, Xiaomeng Chu, Guoliang You, Wuyang Zhang, Daxin Jiang, Yanyong Zhang·· 2 天前AI 评分39
Φ-Bench：大语言模型能否工程化支撑自身运行的基础设施？
$\Phi$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?
AI 导读
研究者发布 Φ-Bench，一个基于真实 LLM 训练与推理代码库构建的基准，含 85 项任务，覆盖九类 LLM 基础设施，任务形式从 kernel 函数补全延伸到长周期实现与仓库级端到端优化。在八个前沿模型中，表现最强的 Claude Opus 5 仅得 36.53%，且所有模型在长周期仓库级实现上的表现都差于 kernel 级任务。该基准将开源。
正文 · 原文
Authors:Leilei Ding, Shumin Wang, Yuting Huang, Fanqi Wan, Yinmin Zhang, Qi Han, Yiming Xu, Feiyuan Zhang, Xiaomeng Chu, Guoliang You, Wuyang Zhang, Daxin Jiang, Yanyong Zhang
Abstract:Large language models (LLMs) now show strong reasoning and coding abilities, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. Existing benchmarks, however, mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and cannot evaluate open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present $\Phi$-Bench, a benchmark of 85 tasks grounded in real LLM training and inference codebases. The tasks cover nine categories of LLM infrastructure and come in three formats, ranging from kernel function completion to long-horizon implementation and repository-wide end-to-end optimization. They are built by a construction pipeline that turns real pull requests, issues, and repository code into executable tasks with validated tests, and every reward is gated by correctness and anti-hacking checks. Across eight frontier models, the strongest model, Claude Opus 5, scores only 36.53%, and every model performs worse on long-horizon, repository-level implementation than on kernel-level tasks. Trajectory analysis further suggests that stronger models treat optimization as controlled experimentation, validating ideas with cheap local runs and ruling out confounders before drawing conclusions. We will open-source $\Phi$-Bench to facilitate future research on LLM agents' ability to develop and optimize the LLM infrastructure that powers themselves.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.10226 [cs.CL] |
| (or arXiv:2609.10226v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.10226 arXiv-issued DOI via DataCite |
Submission history
From: Yuting Huang [view email]
[v1]
Wed, 9 Sep 2026 14:23:47 UTC (2,390 KB)
[v2]
Thu, 15 Oct 2026 08:37:54 UTC (2,338 KB)
来源：arXiv cs.CL · arxiv.org