强智能体做自主 ML 工程需要多少框架？Apple 研究对比最小化编码智能体与复杂框架
How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?
Apple 与 EPFL 的研究者在同等时间预算和相同前沿 LLM 骨干下，对比开源 SOTA 框架与仅提供 read、write、bash 原语的最小化编码智能体单次会话，发现前者没有优势，性能主要由骨干模型决定。通过大规模系统性消融实验，作者认为在编码智能体设定下多智能体编排器、专用检索子智能体等机制层变得冗余，围绕强模型手工搭建框架的投入在当前 MLE 基准上回报有限。
AuthorsKirill Brilliantov†‡, Alejandro Hernández-Cano†‡**, Emmanuel Abbé†
Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more. While such harnesses expand, the use of more primitive but improved coding agents—where LLMs have direct access to the execution environment through read, write, and bash primitives—has received little attention in the field. In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance. Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting. We conclude that the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.
- † EPFL
- ‡ Equal contribution
- ** Work done while at Apple
Related readings and updates.
Continual-learning agents are systems of models, harnesses, and memory operating over long multi-session horizons. Evaluating and training them requires interleaving tasks with agent-side events such as session stop and start, crons, and memory consolidation. Yet existing benchmarks and training frameworks schedule only the benchmark’s own events, leaving each benchmark and agent pair to build a custom scheduling loop. We present SCLATE, an…
Interface agents powered by generative AI models (referred to as “agents”) can automate actions based on user commands. An important aspect of developing agents is their user experience (i.e., agent experience). There is a growing need to provide scaffolds for a broader set of individuals beyond AI engineers to prototype agent experiences, since they can contribute valuable perspectives to designing agent experiences. In this work, we explore the…
来源：Apple Machine Learning Research · machinelearning.apple.com