Databricks 为 Serverless 计算推出成本管控功能

At Databricks, our serverless compute usage has grown more than 3x in the past year, and thousands of customers now run mission critical notebooks, jobs, and pipelines on it. As adoption climbs, the question every platform team asks first only gets louder: how do we gain better visibility into serverless costs and keep the cost within our budget?

Databricks customers running serverless SQL and Model Serving already attribute spend, govern access, and monitor consumption without touching a single cluster knob. We are now bringing that same functionality to serverless compute for notebooks, jobs, and pipelines. We are making serverless cost controls work the way platform teams already expect it to.

Serverless moved cost control up a layer

On classic compute, cost control lived in the infrastructure. You capped spend by capping machines: node counts, instance types, autoscaling ranges, and cluster policies that fenced in what users could provision. The controls were required because you had to manage the infrastructure on your own.

Serverless takes those knobs away, on purpose. Provisioning, scaling, and right-sizing infrastructure is the platform's burden now, handled by Databricks, which is a big part of the value. But cost control has to live somewhere, so it moves up a layer. Instead of attaching to machines, it attaches to people and workloads. Who is allowed to run serverless, how fast a workload is allowed to spend, and where the cost lands are the questions that matter now.

Platform teams describe their needs three ways:

Runaway spend: a single unbounded notebook run can spend thousands in a day, because there is no cluster size or autoscaling ceiling to bound it.
The wrong hands: serverless is available to everyone by default, including users who should not be provisioning it, with no entitlement to gate who can run it.
No accountability: teams will not adopt serverless without the tag enforcement they had on classic compute.

Classic compute answered these with three tools: tags to attribute cost, entitlements to control access, and cluster size limits to cap spend. Serverless needs the same three. Two of them exist today, and the third is coming.

The three objectives of serverless cost control

Attribute costs: know which team, project, or user a serverless DBU belongs to.
Control access: decide who can run serverless in the first place, and on what.
Limit spend: bound how much/how fast a workload or a team can spend, before the fact, rather than after.

We will dive into each of these in order.

Attribute costs

Attribution starts with tags. On classic compute you tagged the cluster; on serverless you tag the compute, and every DBU that compute runs carries that tag through to billing.

Serverless Compute Tags: tag a serverless compute object once, and the workloads that run on it are attributed to the right team, project, or cost center, with no separate policy object to maintain.
System tables: every serverless DBU lands in system.billing.usage, broken out by user, by product, and by tag. A leadership-ready "cost by team" view is a single query, and the account-console cost dashboards render it for you without writing any SQL at all.

The same table answers the audit question you will want before you change anyone's access: who is actually running serverless today, and how much are they spending? That is one more query, ranked by consumption, and the full examples live in the system tables documentation.

Availability: Serverless Compute tags for attribution is currently available in Beta.

Observability is already available and customers can already monitor serverless cost with system tables on AWS, Azure, and Google Cloud.

Watch spend as it moves: budgets, alerts, and dashboards

The same system tables that attribute cost also let you track it. Account-console budgets raise email alerts when serverless spend crosses a threshold, and the cost management dashboards let you track consumption by product, workspace, and tag. A SQL alert on yesterday's serverless spend closes the loop within hours.

Monitoring is reactive, though: it reports the expensive day the next morning rather than preventing it. That gap is why the next two objectives exist.

Availability: Monitoring is generally available today. Set up budgets and spend alerts on AWS, Azure, and Google Cloud.

Control access

The first proactive control is deciding who can run serverless at all. That grant is an entitlement: it defines which users or groups can access serverless, just like existing entitlements control who can create clusters or connect to a SQL warehouse. Serverless Compute Access Controls is now generally available and lets you govern serverless with that model: permissions on a compute object, not policies bolted onto every workload.

A serverless compute object is the permission boundary for serverless. Every workspace gets two by default:

Default Serverless Interactive Compute: notebooks and Databricks Connect.
Default Serverless Automated Compute: jobs and Spark Declarative Pipelines.

Permissions on these objects are the two you already know from warehouses: Can Use and Can Manage. If you can reason about who can attach to a SQL warehouse, you can reason about who can run serverless.

The model is safe by default. Every user starts with Can Use, so turning the feature on changes nothing until an admin acts. When an admin does act, the flow is the one you would expect: open the Compute page, go to the Serverless tab, open a default compute object, remove "All Users," and add a specific group. A user outside that group now sees Serverless grayed out in the notebook compute selector, and nothing that was already running is interrupted.

Availability: Generally available today. See the docs for AWS, Azure, and Google Cloud.

Set spend limits

Access control decides whether a workload runs, not how much it can spend once it does. Closing that gap is the third job - Rate Limits – and it lands on the same compute objects you just saw.

Rate limits on automated workloads: per-compute limits, expressed as t-shirt sizes (S, M, L, XL) that bound how much or how fast a job or pipeline can spend rather than only whether it can run. Admins would create custom compute objects and share them per team, giving each team its own limit. This is the classic cluster-policy pattern, ported to serverless.
Shared budgets for interactive (coming soon): a per-session cap alone cannot protect a team's budget, because many "Medium" sessions still add up. Our direction is a shared limit for the group, with a per-session maximum as the guardrail underneath it.

Attribution improves at the same time. A new serverless_compute_id in system.billing.usage attributes spend per compute object, so "cost by compute object" becomes as easy as "cost by team."

One clarification for anyone who knows DBSQL t-shirt sizes: for notebooks and jobs the size is a maximum autoscaling ceiling, not a fixed cluster size. Pick XL for a job that only needs Medium, and it is allocated Medium.

Availability: Rate limits are now available in Beta. Enable rate limits through your Databricks preview settings; see the docs for AWS, Azure, and Google Cloud.

The full picture

Put the three jobs back together:

Attribute cost: native tags on serverless compute plus serverless_compute_id in system tables, usable in budgets and dashboards.
Control access: entitlements on serverless compute objects.
Limit spend: rate limits, shared across a group and bounded per workload.

Get started today

You can start implementing serverless cost controls today. Here is how you can get going:

Turn on tag enforcement: tag your serverless compute so that attribution is in place before anything else.
Stand up monitoring: add cost dashboards on system.billing.usage plus a budget alert on serverless spend.
Audit who runs serverless today: so that you know the blast radius before you change access.
Decide your access posture: who should be able to run serverless interactive, and who should run automated?
Turn on Serverless Compute Access Controls: enable fast, fully managed compute for the groups you choose.

Serverless Compute Access Controls is generally available now. To go deeper on attribution and monitoring today, the system tables (AWS | Azure| GCP) and serverless tagging (AWS | Azure | GCP) documentation is the place to start.