跳到正文
RadarAI· X:Rohan Paul (@rohanpaul_ai)·· 22 小时前AI 评分38

YC 创业公司 Synthetic Sciences 发布 FrontierML 基准,5 个前沿模型均未达强结果线

AI 导读

YC 创业公司 Synthetic Sciences 发布 FrontierML 基准,用于评测 AI 智能体自主开展实证 ML 研究的能力,测试的 5 个前沿模型在其 25 个开放式问题上无一达到强结果线。每个任务中目标被隐藏,智能体需在 Lab0 用 12 小时和 H100 GPU 自行构建进度代理并校准其可信度,最终提交的模型或方法在 held-out 评估上打分。

正文 · 原文

Rohan Paul· @rohanpaul_ai · X·2026-11-19 13:34· 1 小时前AI 评分51

AI 导读

YC 创业公司 Synthetic Sciences 发布 FrontierML 基准,评测 AI 智能体在自主实证 ML 研究上的能力,作者称测试的 5 个前沿模型无一在其 25 个开放式问题上达到强结果线。每个任务中目标被隐藏,智能体在 Lab0 获得 12 小时和 H100 GPU,需自行设计进度代理并校准其可信度,最终提交的模型或方法在 held-out 评估上打分。

正文 · 原文

Synthetic Sciences, a Y Combinator startup, has launched FrontierML, a benchmark so hard that none of the 5 frontier AI models it tested hit the strong-result bar on any of its 25 ML research problems.

They hide each objective, so agents must build their own tests and decide how much to trust them.

This is a test of whether AI agents can do machine learning research on their own without ever seeing the score they’ll be graded on.

Each agent gets 12 hours and real GPUs, and has to work out for itself how to measure progress.

FrontierML runs each task as a loop with the objective kept out of reach:

The agent gets an open-ended ML problem and 12 hours in Lab0
It designs a local proxy for progress
It runs experiments on H100s and calibrates how far that proxy can be trusted
It submits a model or method
FrontierML scores it on a held-out evaluation

Synthetic Sciences argues step 3 is where much of empirical research happens. Here, a lot of the score rides on it.

引用Synthetic Sciences (YC W26)@SynScience

Introducing FrontierML, a benchmark for evaluating AI agents on autonomous empirical ML R&D.

Most existing benchmarks for ML "research" give the agent a validation set or verifier that closely tracks its final score. In practice, human researchers rarely have that. Much of empirical research is deciding how to measure progress on a problem and how far to trust those measurements.

In FrontierML's 25 open-ended problems, agents have to operationalize progress through their own experiments, constructing local proxies and calibrating how far to trust them, while working toward an objective they never directly observe. Their submitted model or method is then scored on a held-out evaluation.

在 X 查看被引用的帖子

来源:Rohan Paul · x.com

来源:RadarAI · aihot.news