Databrick Serverless compute plane networking

Here’s a simple line-by-line summary of the Serverless compute plane networking (08/04/2025):


General idea

  • Serverless compute runs in a separate compute plane managed by Databricks.
  • You can configure secure connections between this compute plane and your own resources.
  • Databricks charges networking costs when serverless connects to your resources.
  • Control plane ↔ serverless plane always uses the cloud provider’s private backbone (not public internet).

Serverless egress control (outbound connections)

  • Lets you control outbound traffic from serverless compute.
  • Benefits:
    • Security → prevent data leaks.
    • Control → allow only specific storage accounts, domains (FQDNs), or connections.
    • Simpler management → define and manage policies in one place.

Network Connectivity Configuration (NCC)

  • NCC = an account-level, regional configuration to manage networking for serverless.
  • Admins create NCCs in the account console.
  • One NCC can be attached to multiple workspaces.

What NCC enables:

  1. Resource firewalls with service endpoints
    • Databricks provides stable service subnets.
    • These subnets can be added to your Azure resource firewalls.
    • Ensures secure serverless access to storage and other resources.
    • Firewall rules get auto-added to workspace storage accounts.
  2. Private Endpoints
    • You can add private endpoints in an NCC.
    • Databricks then creates a private endpoint request to your Azure resource.
    • Once accepted, the serverless compute plane connects privately (not public).

Extra note

  • Databricks may use service endpoints, private IPs, or public IPs depending on resource type/location.
  • All methods are supported unless specifically stated otherwise.

👉 In short:
Serverless networking lets you securely connect compute to your Azure resources using firewall rules + private endpoints, while controlling outbound traffic with egress policies.


Related Posts

Z-ORDER in Databricks: Magic or Myth?

When it helps, when it hurts, how to pick columns, and what KPIs to measure. Introduction: Why the Hype Around Z-ORDER? If you’ve spent time in Databricks,…

Read More

Taming Small Files in Delta Lake

Optimal file sizes, compaction strategies, and how to keep your Delta tables lightning-fast Why Small Files Are a Big Problem Delta Lake is powerful, but it inherits…

Read More

Databrick Serverless compute limitations

Here’s a simple, line-by-line summary of the Serverless compute limitations (09/29/2025): General limitations Streaming limitations Machine learning limitations Notebook limitations Job limitations Compute-specific limitations Caching limitations Hive…

Read More

Best practices for Databrick serverless compute

Best practices for serverless compute Big picture Before you migrate Ingesting data (getting data in) Querying external data (without moving it) Spark configurations Watch your costs Quick…

Read More

Databrick Serverless compute release

Here’s a simple line-by-line summary of the important points from the Serverless Compute release notes (09/24/2025): Perfect 👍 Here’s a one-page cheat sheet table for the Serverless…

Read More

Shuffle Without Tears in Databricks

How to detect skew, leverage AQE, use repartitioning patterns, and tune the shuffle service for blazing-fast jobs Why Shuffle Is a Big Deal In Spark (and therefore…

Read More
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x