Databricks is constantly evolving, introducing features that make data ingestion, sharing, and collaboration more seamless. While you may be familiar with the basics of loading data into Databricks, there’s a suite of advanced capabilities that take ingestion and interoperability to the next level.
In this article, we’ll explore four key areas:
- Lakehouse Federation
- Zerobus (Coming Soon)
- Delta Sharing
- Databricks Marketplace for Data Ingestion
1. Lakehouse Federation
Lakehouse Federation allows you to query external data sources without moving your data into Databricks.
Why it’s useful:
- Ad hoc reporting without full-scale ETL.
- Quick proof-of-concept projects.
- Exploratory analysis for new pipelines or reports.
- Supporting workloads during incremental migration.
Supported Sources Include:
- MySQL
- PostgreSQL
- SQL Server
- Snowflake
- Amazon Redshift
- Google BigQuery
With Lakehouse Federation, you can work across multiple systems in real time, avoiding costly data duplication.
2. Zerobus (Coming Soon)
Zerobus is a Lakeflow Connect API designed to write event data directly to your lakehouse with:
- High throughput: Up to 100 MB/s.
- Near real-time latency: Under 5 seconds.
Best for:
- IoT sensor streams.
- Clickstream analytics.
- Application telemetry.
- High-frequency event logging.
By simplifying event ingestion, Zerobus reduces the complexity of integrating streaming data sources.
3. Delta Sharing
Delta Sharing is an open protocol that enables secure data sharing across:
- Platforms – Share with Snowflake, Redshift, BigQuery, etc.
- Clouds – AWS, Azure, GCP.
- Regions – Without data duplication.
With Delta Sharing, your data products remain up-to-date and accessible without creating unnecessary copies, maintaining governance through Unity Catalog.
4. Databricks Marketplace – Ingesting External Data
The Databricks Marketplace is an open exchange for all types of data products, powered by Delta Sharing.
Available Assets:
- Datasets
- Notebooks
- Dashboards
- ML Models
- Solution Accelerators
How to Access Data from the Marketplace:
- Navigate to Marketplace in Databricks.
(Left sidebar → Marketplace) - Find Your Assets – Search for datasets, models, or dashboards.
- Get Instant Access – Many assets are free and available immediately.
Example: You could import a simulated retail dataset directly into your workspace for testing ETL pipelines, training ML models, or building dashboards.
Why These Features Matter
These capabilities bridge gaps between data sources, platforms, and teams:
- Federation avoids unnecessary data duplication.
- Zerobus promises near real-time ingestion at scale.
- Delta Sharing enables secure cross-platform collaboration.
- Marketplace accelerates project startup with ready-to-use assets.
