Databricks 基准测试：Genie 搭配 Genie Ontology 准确率 91%，token 用量不到对手一半

Databricks：Blog·2026-11-11 23:00· 1 小时前AI 评分41

Genie with Genie Ontology Delivers 1.6× the Accuracy with Less than Half the Tokens

AI 导读

Databricks 与 Turing 合作构建企业数据智能体基准，在实时 Databricks 工作区中对比 Genie 与两个前沿智能体，Genie 任务通过率 91%，另两者分别为 45% 和 58%。

正文 · 原文

We developed a new enterprise data-agent benchmark in collaboration with Turing to compare Genie with two major frontier agents and their respective harnesses on business questions in a live Databricks workspace. The evaluation measures answer accuracy, token usage and time to complete the task. Genie had a 91% pass rate, compared with 45% and 58% for the other frontier agents, while using fewer than half the tokens.

Unlike previous data agent benchmarks that typically only include tables, this benchmark also includes notebooks, dashboards, saved SQL queries, documents, and other analytical assets that capture enterprise knowledge, business definitions, and calculation rules. These assets provide key enterprise context that table schemas alone cannot establish. Genie Ontology is the enterprise context layer that gives every AI agent a shared understanding of the business. It automatically extracts and organizes this knowledge, surfacing relevant context for Genie to use when answering questions.

Our previous evaluation of Genie with ontology used a proprietary dataset. This benchmark evaluates Genie using tasks and supporting analytical assets designed for sharing, so others can also run the evaluation.

Answering business questions correctly requires accessing the institutional knowledge about how an organization interprets and uses its data. Teams develop this knowledge through everyday analytical work: establishing revenue definitions, maintaining dashboards that reflect current reporting rules, and writing queries that handle business-specific exceptions. The questions in this benchmark require agents to access and reason about this knowledge to perform the tasks correctly.

Why a new benchmark

There are already many existing data agent benchmarks, but none that truly emulate real-world usage. We needed a benchmark that tested agents' ability to discover and use business knowledge distributed across notebooks, dashboards, saved SQL queries and other documents.

Existing data agent benchmarks mostly focus on a set of tables and the pipeline code that builds them, and the performance on these benchmarks is similar between the agents. In real-world enterprise tasks, it’s seldom the case that the correct answer comes from the tables alone. It requires leveraging information from many different sources, such as a specific notebook that builds the reporting tables, the SQL behind a team's dashboard, or saved queries. The questions in this benchmark require the agent to reason about the knowledge residing in these assets and combine them to correctly answer the questions.

Results

For this benchmark, we developed a fictional company called Calderwick Energy, a UK electricity and gas supplier, whose challenges closely resemble those of many typical data teams. There are hundreds of tables (the largest has around 120 million rows of half-hourly meter data), along with assets that different teams have built on top of them, such as notebooks, scheduled jobs, dashboards, saved SQL queries, code, metrics, documents, and Genie Agents. It also has the messiness of a real-world scenario, like copied notebooks, abandoned analyses, old documents and dev copies, frozen reporting tables and dashboards nobody has touched in months.

We created multiple data analytics questions and as queries as the benchmark tasks and ran each agent 5 times per task against the same live Databricks workspace.

We compare the agents on three primary metrics:

1. Accuracy (the percentage of responses that satisfy the task’s grading criteria)
2. Token usage (the average number of tokens used per run)
3. Wall time (the average amount of time taken per run)

The Table below shows the overall performance of the three agents on this benchmark. Genie solves 91% of the tasks correctly, compared to 45% for frontier Agent A and 58% for frontier Agent B. Moreover, Genie is significantly faster (4x compared to Agent A and 3x compared to Agent B) and much more token-efficient, as it can leverage the Genie Ontology to identify and reason about relevant enterprise context more efficiently.

You can read more about how Genie Ontology works in the GA announcement blog or in the official documentation.

If you want to replicate this benchmark on your own workspace, reach out to your account team, and they can help you set it up to help implement this and show how well Genie works on a live Databricks workspace.

Evaluation methodology

In addition to Genie, we evaluated two leading coding/knowledge agents, “Agent A” and “Agent B,” from frontier AI labs in our results. Genie ran natively, while the other agents connected through a Databricks toolbelt. They all received the same question. The toolbelt specifically excluded the Genie MCP because it would give those other agents access to all the Genie capabilities we were testing.

We checked whether each agent answered correctly using a calibrated grading rubric that captures the desired answers and agent behaviors. Where small differences in the numbers were acceptable, we set those limits in advance. An answer passed only if it met the required checks in the grading rubric. We also recorded how the agents found and used the business context needed to answer the question. The traces show which snippets Genie retrieved and the queries the agents chose to run. In addition to accuracy, we also measure latency and token usage for each agent.

From the results, we observe that Genie’s correct answer pass rate is significantly better than both frontier agents, Agent A (+46%) and Agent B (+33%). It is also significantly faster (3-4x) while using less than half the tokens on average. We now provide a qualitative analysis of how Genie efficiently answers these questions correctly.

Example Walkthrough: Finding the right planning rate

The following is an example task in the benchmark that asks agents to identify one of Calderwick Energy’s historical Commercial planning rates:

This looks like a straightforward lookup. The bill cost stack table has a column called cfd_rate_p, and the rows for FY24/25 contain 0.31 pence per kWh. An agent can find the relevant table, filter to the requested period, and return that number. Checking more rows would reinforce the answer: the same rate appears throughout the table.

The detail that changes the answer is already recorded in the notebook that builds the bill cost stack.

It explains that Commercial sets planning rates annually, and that the mart uses FY25/26 assumptions, including when those assumptions are applied to earlier billing periods. A separate section, “Planning rates by year,” preserves the history:

The question asks which planning rate the Commercial team set for FY24/25. The rate currently applied to those rows reflects FY25/26 assumptions. The notebook already documents this distinction and preserves the historical rate needed to answer the question.

Genie Ontology surfaced a snippet containing the historical planning rates and linked it to the source notebook, 45_mart_bill_cost_stack. When we ran the task, Genie inspected the notebook to confirm the rate and returned 0.52 p/kWh in its first response. The Ontology panel showed both the relevant information and the notebook it came from, allowing us to check the source.

Across the 5 Genie runs, the first answer was 0.52 every time. The other agents did not get the correct historical rate across their runs. One returned 0.31 throughout. The other reported 0.31, questioned whether a year-specific planning rate was documented, or both.

Answering correctly required understanding what the rate in the table represented and finding the historical planning assumptions documented by the Commercial team. Genie Ontology made that existing knowledge available as context for Genie’s response.

The cost vs. quality trade-off

The frontier agents are very good at doing most things based on written text. Type the words for a task that can be done with a computer, and an LLM will understand the requirement and then create and execute a plan to complete that task. Each new model release adds incrementally more capability, and the frontier labs seem to swap out the No. 1 spot on the various benchmark tests.

This effectively commoditizes the token-generating ‘intelligence’ and puts the source of the power and capability of the agents into the harness and the context they can retrieve. This is why this benchmark shows how much context matters and how Genie Ontology is optimized to provide the relevant context for the task at hand. For a competing agent, retrieving all the artifacts and metadata needed to extract the relevant context is time-consuming and expensive, and it does not necessarily guarantee the correct result. These benchmark results make the case for using Genie (both the UI and the MCP) and Genie Code as the most reliable and cost-effective data agent and the entry point for your AI coworker.

The benchmark is intended to test Genie and other agents operating within an established Databricks deployment, with relevant data governed by Unity Catalog and business knowledge captured in workspace assets. Agents need to discover and use the correct context. In new deployments that have not had time to accumulate workspace knowledge, or when questions can be answered directly from tables and schemas, the difference between agents may be smaller. This explains why other existing data agent benchmarks show smaller differences between agents.

Learn more

To learn more, reach out to your account team to get access to this benchmark, and they can help you set it up to help showcase how Genie works on a busy Databricks workspace.

Genie Ontology is now Generally Available. For more information:

• Visit the Genie Ontology product page
• Read the product documentation
• Join us for the Data + AI Live: Context is just the beginning event with Ali Ghodsi and Greg Brockman
• Download the eBook: Operationalizing the Enterprise Context Layer

This benchmark is just a part of our Week of Genie, with more news coming on Genie, Genie Agents, and Genie MCP, all powered by the shared business context of Genie Ontology.

We're excited to see how customers use Genie Ontology to scale trusted AI across their organizations.

来源：Databricks：Blog · databricks.com