Databricks Lakehouse Federation：不迁移数据也能做投资分析

Databricks：Blog·2026-10-28 02:00· 1 小时前AI 评分42

Unlocking Investment Insights Without Migrating Your Data: Lakehouse Federation

AI 导读

Databricks Lakehouse Federation 支持通过标准 SQL 直接查询 SQL Server、PostgreSQL、MySQL、Snowflake 等外部数据库，数据留在原处，无需复制或同步。

正文 · 原文

Lakehouse Federation: Query Anywhere, Move Nothing

In today's enterprise landscape, data rarely sits in one place. Financial institutions, asset managers, and individual investors face a common challenge: their most sensitive data including portfolio holdings, trading positions, and client information often resides in on-premises databases behind corporate firewalls, in the cloud, or across both environments. That challenge is compounded by the fact that the third-party data they need for analysis, including market data, analyst estimates, and financial fundamentals increasingly lives in the cloud.

Traditional approaches force an uncomfortable choice: copy sensitive data to the cloud, stand up brittle ETL pipelines, or give up on integrating the two altogether.

What if you could just federate the on-premises source, subscribe to the third-party data on the cloud, and go straight from those two catalogs to a dashboard and a natural-language chat, with no pipeline, no materialized views, and no persisted objects in between?

That's Databricks Lakehouse Federation at its simplest.

The Solution: Query Anywhere, Move Nothing, Build Nothing

Databricks Lakehouse Federation lets you create federated connections to external databases like SQL Server, PostgreSQL, MySQL, Snowflake, and more, and query them directly from Databricks using standard SQL. The data stays exactly where it is. No copying. No syncing. No compliance headaches.

Add a marketplace data share on top of it, and you have two catalogs sitting side by side in Unity Catalog - one federated, one shared, that a plain SQL query can join like they were any two ordinary tables.

A Real-World Example: Client Portfolio Analysis

Let's walk through a concrete example. Imagine you're a wealth management firm with:

• First-party data: Your portfolio holdings - which stocks you own and how many shares, stored in an on-premises SQL Server database.
• Third-party data: FactSet financial data from the Databricks Marketplace, including earnings fundamentals and analyst estimates.

Your goal: Build an investment dashboard that shows your projected earnings, identifies high-risk positions, and recommends actions, all without exposing your holdings data to the cloud.

Step 1: Set Up Access to Portfolio Data and FactSet Data

Step 1a: Set Up the Federated Connection

Setting up federation is straightforward through the Databricks UI. Navigate to Catalog → External Data → Connections and create a new connection to your on-premises database.

The connection securely stores your credentials.

Step 1b: Create a Foreign Catalog

Once the connection is established, create a foreign catalog that maps to your on-premises database. This makes your external tables appear as native Unity Catalog objects.

After this step, your on-premises data appears as a catalog, but the data never leaves your SQL Server. Every query executes against the source in real time.

Step 1c: Subscribe to FactSet Data from the Marketplace

The Databricks Marketplace provides instant access to FactSet's comprehensive financial data. Search for FactSet, review the available datasets, and request access. Once the provider approves your request, you can subscribe to the relevant datasets.

After subscribing, the data is available as a shared catalog in your workspace, organized into schemas:

• fe_v4 - FactSet Estimates: analyst consensus EPS forecasts, earnings projections, and estimate counts
• ff_v3 - FactSet Fundamentals: annual financial statements including revenue, net income, profit margins, and debt levels
• sym_v1 - Symbology: ticker-to-entity mappings that connect your portfolio symbols to FactSet's internal identifiers

These tables appear alongside your federated portfolio catalog in Unity Catalog. You can browse them, preview data, and check schemas, all without any ingestion or data copying.

Step 2: Explore the Combined Data

Step 2a: Explore the combined data with SQL

With both data sources accessible through Unity Catalog, an asset manager can open the SQL editor and write one query that spans both the federated holdings on one side, the FactSet marketplace tables on the other. To the SQL author they look like two ordinary tables in the same warehouse.

There is no pipeline to author, no materialized view to schedule, and no gold layer to maintain. Every time the query runs it re-federates the on-premises source live and re-joins it against the marketplace share.

Step 3: Consume the Data

Step 3a: Create the Investment Dashboard

The dashboard is powered by the same combined query inlined directly into each widget's dataset. No views are created and no tables are persisted. Every widget re-federates the source on each refresh.

It instantly answers questions like:

• Portfolio health: total holdings, current annual earnings, projected annual earnings, high-risk count
• Risk distribution: what share of the book is high / medium / low risk?
• Investment recommendations: how many stocks should I buy more of vs. sell?
• Top performers: which holdings are generating the most earnings and how do current earnings compare to projected earnings?
• Full holdings table: per-stock earnings, growth %, margin %, D/E, risk, and action recommendation.

Step 3b: Enable Natural Language Queries with Genie

The final piece: a Genie agent wired directly to the same federated holdings and FactSet marketplace tables. Genie receives the join chain and data-quality filters in its instructions. It translates natural language into the same queries the dashboard runs.

The result: democratized data access. Portfolio managers, financial advisors, and individual investors can all get answers without writing SQL, and without anyone materializing anything.

The Complete Picture

Databricks Lakehouse Federation fundamentally changes the economics of hybrid data analytics. Instead of the expensive, risky, and time-consuming process of moving data to the cloud, or the equally burdensome process of building and maintaining pipelines on top of it, you can bring the compute to the data and let a single query drive everything downstream.

In this example, the walkthrough built a complete investment analytics solution that:

• Queries on-premises holdings data in real time
• Enriches it with FactSet financial intelligence from the Marketplace
• Combines both in a single SQL query, with no persisted intermediate objects
• Visualizes everything in an interactive dashboard
• Enables natural language queries with Genie

All without copying a single row of portfolio data to the cloud.

The future of data analytics isn't about moving all your data to one place and it isn't about building elaborate pipelines to reshape it either. It's about querying data wherever it lives. While this blog used investment portfolios as the example, this pattern – federate sensitive first-party data, subscribe to cloud-based third-party data, and analyze without moving or copying anything – applies broadly. Healthcare organizations can join HIPAA-protected patient records with drug interaction databases for treatment recommendations.Manufacturing firms can federate on-premises IoT telemetry with cloud-based supply chain data. Retailers can combine in-store POS systems with third-party demographic datasets.The common thread: sensitive first-party data stays protected while enabling rich analytics with third-party data.

Ready to get started? Explore the following links:

• Set Up Federation — Follow the Lakehouse Federation documentation to connect your external databases
• Browse the Marketplace — Browse available data products at marketplace.databricks.com
• Create Dashboards — Use AI/BI dashboards to visualize insights
• Enable Genie — Let your users ask questions in natural language
• View the full code on GitHub — Code for this blog

来源：Databricks：Blog · databricks.com