斯坦福与Google DeepMind用AlphaEvolve进化工具调用基准

Rohan Paul· @rohanpaul_ai · X·2026-11-09 07:36· 57 分钟前AI 评分46

AI 导读

斯坦福与Google DeepMind提出TURTLE方法，用AlphaEvolve对AppWorld等静态工具调用基准的现有任务做变异，加入过滤条件、额外步骤和隐藏查询，把过易的基准改造成仍可解但更难的测试。

正文 · 原文

New Stanford and Google DeepMind paper shows that you can turn too-easy tool-use benchmarks into hard, still-solvable tests by evolving existing tasks against the model.

Tool-use benchmarks like AppWorld are static, so models soon solve most tasks, and building new ones takes human work.

TURTLE uses AlphaEvolve to mutate existing tasks, adding filters, extra steps and hidden lookups, then runs the solver model 10 times on each. Tasks score highest when the solver succeeds about 50% of the time, and each must be solved at least once.

69% of original AppWorld tasks were always solved by Gemma 4 31B-it, while nearly all 735 evolved tasks sat at 40% to 60% completion. An LLM judge rated 52.5% of evolved tasks hard, versus 19.8% of the originals.

If your model aces a benchmark, evolve its tasks toward a 50% success rate instead of writing a new benchmark from scratch.

来源：Rohan Paul · x.com