Bright Machines 如何在不开放工厂入站访问的情况下提供实时工厂数据看板

Databricks：Blog·2026-11-12 00:10· 51 分钟前AI 评分44

Air gapped, not data dark: how Bright Machines gives customers live factory dashboards without opening the plant

AI 导读

Bright Machines 用 Debezium 在工厂边缘捕获 PostgreSQL 变更、写成 Apache Iceberg，经单一可审计出站路径推送，再由 Databricks 通过 Unity Catalog federation 原地读取，全程不开放任何入站端口。

正文 · 原文

Bright Machines gives customers live, governed dashboards of their factory data without opening any inbound access to its air-gapped plants. It captures PostgreSQL changes at the factory edge with Debezium, writes them as Apache Iceberg, sends them out over a single auditable outbound path, and reads them in place in Databricks through Unity Catalog federation. The first customer-visible dashboard went live about two months after kickoff.

Problem: customers wanted factory data, and the only way to share it was reports rerun on-prem and sent over by email.
Solution: Debezium change data capture, Apache Iceberg, AWS Glue, Unity Catalog, Databricks SQL, AI/BI Dashboards and OpenSharing.
Result: nine loosely coupled services replaced by one governed platform plus one edge path, with zero inbound ports to the plant.
Freshness: a row committed on the line is queryable in Databricks in one to two minutes, and dashboards refresh every ten minutes.
Monitoring: a heartbeat check detects a broken pipeline in under thirty minutes, down from days.

Why couldn't customers see their own factory data?

Every time a customer visited one of our factory floors, the same thing happened. They would stand in front of the assembly dashboard, watch the line run, and ask for a copy of the data: yield by station, cycle time trends, downtime by cause, the genealogy of a batch.

We had all of it. We had built those dashboards for our own operations team, and they ran on the floor against the line databases. A customer could not access any of it directly over the internet. The plant is air gapped by design: no inbound access, one tightly controlled route out, and role-based access rules that restricted access in.

Each new report required data engineering to reshape data through multiple ETL jobs, costing hours to days of engineering time.

Who is Bright Machines?

Bright Machines builds software-defined, automation-first production lines that run inside our customers' plants. Those plants are locked down by design: no inbound access, minimal outbound, and zero tolerance for risk. Each line runs a stack of on-premises services (our in-house MES, integrations with customer's MES, operator screens, robot workflow orchestrators, and the assembly dashboards) that all persist to PostgreSQL on the factory's on-prem cluster. The software team that runs this is a lean team and manages the data pipelines end to end as well as the data applications software stack. The data drives various KPI dashboards, finance reporting, operations, quality, vendor component quality, NCR tracking, genealogy, OEE and the downtime and cycle-time analysis behind every shift.

What did the old factory data pipeline look like, and why did it fail?

On our first attempt to solve this problem, we rebuilt a cloud copy of each factory database from application events, across nine loosely coupled services. On the factory floor the picture was fine. Everything off the floor was the problem with this complex architecture and pipeline.

The nine services were a message broker, a custom shoveler, a queue, object storage, few serverless functions, a cloud relational database, a scheduler, an ETL pipeline, and a popular open source BI tool with very basic RBAC support.

The pipeline failed for these reasons:

Schema Evolution. Schemas changed often, and the pipeline silently dropped changed data unless the ETL jobs were updated every time an on-prem dashboard or schema changed.
Missing monitoring. The pipeline had failures and the dead-letter queues were a pain to monitor, so missed or duplicated events went unnoticed.
Data validation. Once data landed in a bucket, nobody could say with confidence if it was accurate and from the right source.
Data Governance. Having proper governance and role based access controls over views and data that we shared.

Fixing all the above features needed rearchitecting from the ground up.

What were the requirements for the new architecture?

In addition to the above reasons these constraints also shaped the new design:

Security Model. No inbound connectivity to the plant, and all access auditable.

Governance we could demonstrate. A customer would see exactly what their role allowed on the floor, and we had to be able to prove the appropriate limits were in place.

Freshness in minutes. A dashboard view that is on prem would be visible in a few minutes over the internet with the same data.

The new architecture

Why did Bright Machines choose edge capture and federation over direct ingestion?

Bright Machines chose edge change data capture with Unity Catalog federation because it was the only option where no compute outside a customer's and/or Bright Machines' own accounts touched factory data before governance is applied.

Our criteria, in order: access control that matched what was already enforced on the floor, one outbound auditable path we could explain to any security team, minute-level freshness, and open formats at the edge.

We worked through the options with our Databricks solutions architect before writing any code. The two obvious ones were Lakeflow Connect ingesting directly from PostgreSQL, and a Postgres export picked up by Auto Loader. Both would have been faster to stand up. Our threat model followed the rule that data can be pushed out, but nothing outside can reach in. A pull-based ingestion pattern needs a network path from the cloud to the factory database, and running ingestion in an environment outside the accounts where the data resided. So the question became: how do we get the same governed result while keeping every piece that touches raw factory data inside environments we own?

The answer was to flip the direction. Capture changes at the edge, land them in an open format, push them out over one auditable path, and let Databricks read them in place.

We proved it on a replicated line, similar to that of a production site, with two engineers over about six weeks. The bar had two parts. First, a row committed on the line had to be queryable in under five minutes with no inbound port opened. Second, a customer principal had to see exactly the tables and dashboards their role allowed, and nothing else.

The deciding factor was that Databricks already enforced the security model that aligned with ours: Unity Catalog for who can see what, and OpenSharing to publish governed tables without copies. Going in, we worried that governance on federated tables would not be strict enough for the audit. It was strict, and the strictness that tripped us up during cutover is the same strictness we rely on now.

How does data get from an air-gapped factory into Databricks?

Data leaves the factory in five steps, over a single outbound path, and Databricks reads it in place without a copy job. Operators keep the on-prem assembly dashboard, because they need real time. Databricks cloud serves customers and internal teams.

Capture at the edge with Debezium. Debezium Server tails the PostgreSQL replication slot and writes an append-only Apache Iceberg change log to a local object store. If the WAN drops, the backlog becomes objects on the floor, not write-ahead log piling up on the production database.

Send it out over one path. A single edge component copies each object to a bucket in the site's cloud account. It is the only outbound data path from the plant, and its credentials can write to that one bucket and nothing else. Cross-account replication carries the objects to the analytics account.

Federate it into Unity Catalog. The Iceberg tables are registered in AWS Glue, and Unity Catalog federates Glue as a read-only catalog. That makes over a hundred tables across factory PostgreSQL databases queryable in Databricks with no copy.

Model it in Databricks SQL. Bronze views collapse the change log to the current state. A Lakeflow Job rebuilds silver models every ten minutes, all in SQL and deployed from Git with Databricks Asset Bundles.

Serve it with dashboards and OpenSharing. Multiple AI/BI Dashboards read processed data from the silver layer with role-based permissions. This allowed external customers and internal teams to all see the same numbers. OpenSharing allows the same copy of data to be shared securely to our customers.

Why register Iceberg tables in AWS Glue instead of pointing Databricks at the bucket?

Registering the tables in AWS Glue lets Bright Machines control exactly when each table moves forward. A change log on object storage is a stream of new metadata files. A small function advances each table's pointer in Glue only when a new metadata file has fully replicated, and only forward. Glue holds the pointer, Unity Catalog holds the permissions, and Databricks only ever reads the version we have validated.

How do you tell an idle production line from a broken pipeline?

Use a heartbeat. When a line stops for a shift change, no new rows arrive, and that looks exactly like a broken pipeline. Row counts cannot tell the two extremes apart.

Debezium updates a one-row heartbeat table in each source database every five minutes, whether or not the line is running. That update travels the same path as production data. The first task of the Lakeflow Job checks the heartbeat, and if any is more than thirty minutes old, the task fails and the whole job goes red. Dashboards keep showing the last good data instead of presenting stale data as current.

How was the migration phased, and what did we learn?

We phased it in four steps: 1.) a pilot on a PoC line, 2.) going live with the highest-value production line, 3.) a parallel run against the old snapshot path to check row for row parity, and 4.) a hard cutover of the dashboards for customers.

What surprised us. Scale showed up in the wrong dimension. We moved a few 100 gigabytes a day, but as hundreds of thousands of small objects, so every design decision ended up being about object rate rather than dictated by bytes.

What we would do differently. Build the reconciliation pass on day one. A sync driven by create events never goes back for an object it missed, and nothing downstream notices until a query touches it.

What results did Bright Machines see?

The focus of those customer visits changed. Nobody asks for a copy of the dashboard any more, because they already have it, under the roles available to them. A customer analyst opens the Quality dashboard from their desk and sees the same numbers the line lead sees on the floor, within about ten minutes of them happening. We never run a report. The first customer-visible dashboard went live about two months after kickoff.

What should other teams learn from this?

Five lessons for anyone sharing data from a locked-down site:

Start from the security model you already have. Our data governance requirements defined the roles and restrictions. We picked the platform that enforced exactly those, rather than approximating them.
Design the one path out first, and buffer behind it. A single auditable egress makes every later conversation with a security team manageable. Buffering at the edge means a WAN blip never becomes a production database incident.
Keep control of when a table moves forward. A catalog hop you own, with a guard on the pointer, means the lakehouse only ever sees a version you have validated.
Build the data once and let permissions do the rest. Maintaining a customer version and an operations version of every report was what ate the hours.
Assert freshness on a heartbeat, not on row counts. No new rows means the line is idle. No new heartbeat means the pipeline is broken.

What's next for Bright Machines?

Now that the data is governed and accessible, the next step is to make this more accessible to AI features through natural language analytics with Genie, custom AI agents for operations, and predictive analytics that turn factory data into forward-looking decisions.

Having custom AI agents for operational analytics and investigations. The people who work on those lines - operators, quality engineers, maintenance techs have domain expertise but aren't SQL users. Custom AI agents (via Databricks Apps or Agent Bricks) could let a line lead ask "Why did station 4 have higher downtime this shift?" Bright Machines provided knowledge docs along with operational data could get an answer grounded in the same governed silver data. This is the "improve the bottom line" play — reducing the time from question to action on the factory floor.

FAQ Section

Can Databricks be used with an air-gapped factory? Yes. Bright Machines keeps zero inbound ports open to the plant for data. Changes are captured inside the factory and pushed out over one auditable outbound path, and Databricks reads the resulting Apache Iceberg tables in place through Unity Catalog federation.

How fresh is the factory data in Databricks? A row committed on the line is queryable in Databricks in one to two minutes at steady state. Silver models and dashboards refresh every ten minutes. Operators who need real time views keep using the on-prem assembly dashboards.

How do customers access the data? In two ways: AI/BI Dashboards with role-based permissions integrated with Bright Machine’s SSO Federation, and additional capability to provide OpenSharing access, which publishes zero copy governed silver tables to customers with Databricks accounts.

What replaced the old nine-service pipeline? One governed platform, Databricks, plus a single edge path made up of Debezium, a local object store and one sync component. Bronze views, silver models, the job, the dashboards and the shares are all deployed from one Databricks Asset Bundle.

This design was specifically built on AWS, does the design work on Azure or Google Cloud? The cloud-specific parts are the edge sync target, the cross-account replication and the catalog hop. Everything on the Databricks side is the same on any cloud, so moving it is a change to a few resources rather than a rebuild.

How long did it take? The proof of concept took about six weeks with two engineers. The first customer-visible dashboard went live about two months after kickoff.

Want to share governed data from locked-down sites without opening inbound access? Learn how Unity Catalog federates Apache Iceberg tables from AWS Glue, or talk to your Databricks account team about your architecture.

来源：Databricks：Blog · databricks.com