Deepnote 发布 Agent Initiative Benchmark，GPT-6 Astra 以 8.97 分居首

Rohan Paul· @rohanpaul_ai · X·2026-11-13 23:24· 22 分钟前AI 评分48

AI 导读

Deepnote 发布 Agent Initiative Benchmark，把主动型智能体放进模拟公司的 Slack 中，考察其是否知道何时介入、说什么、何时保持沉默。

正文 · 原文

This is a beautiful concept for a benchmark.

Deepnote (@deepnotehq), an agent workspace company, released the Agent Initiative Benchmark, a test that drops proactive agents into a company Slack and scores whether they know when to jump in, what to say, and when to stay quiet.

GPT-6 Astra led with 8.97 out of 10, and GPT-6.1 Sol followed at 8.65 for $0.11 per task, against $1.87 per task for third-placed Claude Fable 5.2.

It runs every model through the same loop:

A thread starts in a simulated company’s Slack
The agent decides whether to act at all
It picks which of 11 services to check and investigates
It posts and tags the right people, or stays silent
Two LLM evaluators score the run 0 to 10 against a reference answer

引用Jakub Jurovych@jakubjurovych

What’s the best model for proactive agents?
Today, we’re introducing the Agent Initiative Benchmark. We tested 12 frontier models on realistic workplace tasks inside a synthetic company with 6 years of highly detailed history: it has a financial model, CRM records, past-due invoices, Linear tickets, and real-life events affecting company development - competitor product announcements, SVB collapse, etc.
We used the company’s Slack to test how a proactive agent listening in on each conversation would assist the team in day-to-day tasks: from stepping in to align initiatives across teams to warning the team not to use an unapproved customer logo.
Highlights:
6.1 Sol and Astra from @OpenAIDevs beat Anthropic's latest Fable 5.2, with Sol crushing it at both performance and cost ($0.11 per task vs $1.87).
The most expensive models aren't the best: Fable 5.1 is 430x more expensive than GPT-6 Luna, with similar performance.
Most models try to help even if they're not needed. Interestingly, in these failure cases, they spend more time thinking, and the unnecessary work takes about 80% more time and money than if they stayed quiet.
If you’re building agents, keep reading for more tips on which model is best for your use case. Detailed breakdown, the open-source dataset, and benchmark code below:

在 X 查看被引用的帖子

来源：Rohan Paul · x.com