Z-ORDER in Databricks: Magic or Myth?
When it helps, when it hurts, how to pick columns, and what KPIs to measure. Introduction: Why the Hype Around Z-ORDER? If you’ve spent time in Databricks,…
Taming Small Files in Delta Lake
Optimal file sizes, compaction strategies, and how to keep your Delta tables lightning-fast Why Small Files Are a Big Problem Delta Lake is powerful, but it inherits…
Databrick Serverless compute plane networking
Here’s a simple line-by-line summary of the Serverless compute plane networking (08/04/2025): General idea Serverless egress control (outbound connections) Network Connectivity Configuration (NCC) What NCC enables: Extra…
Databrick Serverless compute limitations
Here’s a simple, line-by-line summary of the Serverless compute limitations (09/29/2025): General limitations Streaming limitations Machine learning limitations Notebook limitations Job limitations Compute-specific limitations Caching limitations Hive…
Best practices for Databrick serverless compute
Best practices for serverless compute Big picture Before you migrate Ingesting data (getting data in) Querying external data (without moving it) Spark configurations Watch your costs Quick…
Databrick Serverless compute release
Here’s a simple line-by-line summary of the important points from the Serverless Compute release notes (09/24/2025): Perfect 👍 Here’s a one-page cheat sheet table for the Serverless…
Shuffle Without Tears in Databricks
How to detect skew, leverage AQE, use repartitioning patterns, and tune the shuffle service for blazing-fast jobs Why Shuffle Is a Big Deal In Spark (and therefore…
Photon vs. Non-Photon: When It Matters
Benchmarking SQL/Delta workloads, common pitfalls, and a practical migration checklist Why Photon Exists Databricks introduced Photon, a vectorized query engine built in C++ and tightly integrated with…
Autoscaling That Actually Saves Money in Databricks
How to set min/max nodes, use termination settings, mix spot/preemptible nodes, and avoid “yo-yo” scaling Why Autoscaling Is Tricky Autoscaling is one of Databricks’ most powerful features—but…
Right-Sizing Clusters Like a Pro in Databricks
How to pick nodes, cores, memory, and disk for ETL vs. ML vs. SQL—and when to scale up vs. out Why Cluster Right-Sizing Matters Databricks gives us…
Lakehouse Federation (Query Federation) in Databricks — What, Why, How
Lakehouse Federation lets you query external databases directly from Databricks—without copying the data into your lake. You point Unity Catalog at a source (PostgreSQL, SQL Server, Redshift,…
What is a streaming table in Databricks?
What is a streaming table in Databricks? A streaming table is a Delta table that Databricks keeps up-to-date automatically as new data arrives. It uses Structured Streaming…
Materialized view vs. (logical) view
Topic View Materialized View What it is A stored query; results are computed each time you query it A precomputed table of the query’s results, stored on…
Data Warehousing on Databricks with Databricks SQL & SQL Warehouses
What is Databricks SQL (DBSQL)? Databricks SQL is the lakehouse analytics experience for running SQL, building dashboards, alerts, and jobs on Delta tables. You connect to a…
Databricks Serverless Compute for Notebooks, Jobs, DLT, ML & Warehouses — Architecture, Setup, and Tips
What “serverless” means on Databricks Serverless lets you run Databricks workloads without creating or sizing clusters. Databricks provisions, scales, patches, and retires the compute for you, so…
Delta Sharing in Databricks — A Practical How-To (Provider & Recipient)
Delta Sharing lets you publish live, governed data to other teams or external partners—without copying files or building custom APIs. Recipients can query the latest data directly…
Workspace‑Catalog Binding in Unity Catalog (Limit Catalog Access by Workspace)
Why workspace‑catalog binding? By default, any workspace attached to the same Unity Catalog metastore can see and access catalogs (subject to object‑level grants). Workspace‑catalog binding lets you…
Column Level Masking in Unity Catalog — Simple, Practical Guide
What is column‑level masking? Column masks hide or transform sensitive values (PII/PCI/PHI) at query time. Every read of a masked column is replaced by the output of…
Row‑Level Filters & Column Masks in Unity Catalog
What you’ll learn Prereqs: Unity Catalog enabled; DBR 12.2 LTS+ for reads (see notes for dedicated compute), and a SQL warehouse or UC‑enabled cluster. Concepts in 30…
Databricks Secret Management & Secret Scopes — the Practical Guide
Secrets (passwords, keys, tokens) should never live in notebooks or job configs. Databricks gives you a built-in, governed place to put them—secret scopes—and safe ways to read…