Delta Sharing lets you publish live, governed data to other teams or external partners—without copying files or building custom APIs. Recipients can query the latest data directly from their tools…
TL;DR Delta Sharing lets you share live, governed data from Unity Catalog with other teams or external partners without copying it. You create a share, add data assets to it…
Why workspace‑catalog binding? By default, any workspace attached to the same Unity Catalog metastore can see and access catalogs (subject to object‑level grants). Workspace‑catalog binding lets you restrict which workspaces…
What is column‑level masking? Column masks hide or transform sensitive values (PII/PCI/PHI) at query time. Every read of a masked column is replaced by the output of a masking function,…
What you’ll learn Prereqs: Unity Catalog enabled; DBR 12.2 LTS+ for reads (see notes for dedicated compute), and a SQL warehouse or UC‑enabled cluster. Concepts in 30 seconds When to…
What you’ll learn Prereqs: Unity Catalog enabled, a catalog/schema you can write to, and a SQL Warehouse or notebook attached to a UC‑enabled cluster. Quick definitions Create your first scalar…
Secrets (passwords, keys, tokens) should never live in notebooks or job configs. Databricks gives you a built-in, governed place to put them—secret scopes—and safe ways to read them at runtime.…
Quick role guide (what each should do) Persona Typical actions Where they work UC privileges (high level) Workspace entitlements Admin / Data Steward Govern data, manage grants, ownership, create catalogs/schemas…
Unity Catalog (how security works) How to secure tables (and “hide” them) (You can also grant at catalog level to cascade.) (Databricks Documentation) What permissions exist? (most-used, by object) What…
1) Add users in Databricks Best practice: sync from Microsoft Entra ID (SCIM). You can also add users manually. UI (manual) CLI (workspace-level) This creates the user in the workspace…
What is Databricks Secret Management? A built-in way to store credentials (keys, passwords, tokens) safely and read them from notebooks/jobs without printing them. Values you fetch with dbutils.secrets.get() are auto-redacted…
What Auto Loader does (one line) Watches a folder for new files and appends them exactly once to a Delta table, even if the job restarts. Minimal example (Python) Replace…
This is the short, practical version of the big guide. Three tools you’ll use again and again: We’ll use one tiny table and show copy‑paste SQL for each feature. Sample…
This guide distills how to get the most out of Databricks Delta tables using three modern primitives: Use these together to build tables that are fast to query, cheap to…
Databricks Auto Loader is the cloudFiles source for Structured Streaming that incrementally discovers and ingests new files from ADLS/S3/GCS into Delta Lake—with built‑in deduplication, schema management, and exactly‑once delivery to…
Databricks COPY INTO is a lightweight, SQL‑first way to load files from cloud storage (ADLS/S3/GCS) into Delta tables. It’s retryable and idempotent—you can schedule it safely without creating duplicates because…
Databricks provides the dbutils module as a built-in set of utilities to interact with Databricks services directly from notebooks.These utilities allow you to: 1. Hierarchy Overview The main structure of…
When working with Databricks, you will store and access data from various locations — some are Databricks-native (like DBFS), and others are external (like Azure Data Lake Storage via ABFSS).Choosing…
1. Introduction As datasets in Delta Lake grow into terabytes or petabytes, performance and storage optimization become critical.Liquid Clustering and Deletion Vectors are two modern techniques in Databricks that solve:…
If you’ve been working with SFTP connections in Databricks using the pysftp library, you might have run into this frustrating error: This typically pops up when your Databricks job or…