Databricks Secret Management & Secret Scopes — the Practical Guide

Mohammad Gufran Jahangir August 17, 2025 0

Secrets (passwords, keys, tokens) should never live in notebooks or job configs. Databricks gives you a built-in, governed place to put them—secret scopes—and safe ways to read them at runtime. (Microsoft Learn)

Table of Contents

What is Secret Management in Databricks?

A secret scope = a named folder of key–value secrets (for example, scope jdbc, keys username, password).
Scopes can be Databricks-backed (encrypted by Databricks) or Azure Key Vault-backed (secrets stay in your AKV).
Secrets you fetch with dbutils.secrets.get() are auto-redacted in cell outputs and logs. (Microsoft Learn, Databricks Documentation, Stack Overflow)

Two scope types (and when to use each)

Scope type	Where secrets live	Best for
Databricks-backed	Encrypted store managed by Databricks	Quick workspace-local secrets, POCs, CI jobs
Azure Key Vault-backed	Your Azure Key Vault; Databricks reads them	Centralized enterprise governance & rotation

Docs: secret scopes overview; AKV requirements and notes. (Microsoft Learn, Databricks Documentation, kb.databricks.com)

Fast path: create a Databricks-backed scope, save secrets, use them

Requires the modern Databricks CLI (and login). See “CLI install & auth” below.

# 1) Create a scope
databricks secrets create-scope my-scope

# 2) Put secrets into it (you’ll be prompted to paste values, hidden)
databricks secrets put-secret my-scope username
databricks secrets put-secret my-scope password

Use in a notebook / job:

user = dbutils.secrets.get("my-scope", "username")
pwd  = dbutils.secrets.get("my-scope", "password")  # prints as [REDACTED] if echoed

Handy tutorial that strings it all together: create scope → add secrets → use in JDBC. (Databricks Documentation)

You can also read secrets directly from SQL:

SELECT secret('my-scope','username') AS user_name;  -- Databricks SQL function

(Databricks Documentation)

Enterprise path: create an Azure Key Vault-backed scope

UI (simplest)
In Databricks → User settings → Secret scopes → Create → choose Azure Key Vault-backed, then paste:

DNS name: https://<kv-name>.vault.azure.net/
Resource ID of the vault

The workspace identity (or your SP) needs Contributor/Owner on the vault to create the scope. Rotate secrets in AKV; Databricks reads them at runtime. (Microsoft Learn, kb.databricks.com)

CLI (scriptable)

databricks secrets create-scope my-akv-scope \
  --scope-backend-type AZURE_KEYVAULT \
  --json '{
    "scope": "my-akv-scope",
    "scope_backend_type":"AZURE_KEYVAULT",
    "backend_azure_keyvault":{
      "resource_id":"/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<kv-name>",
      "dns_name":"https://<kv-name>.vault.azure.net/"
    }
  }'

(Databricks Documentation)

Use exactly the same way in code:

token = dbutils.secrets.get("my-akv-scope", "sp-client-secret")

Installing & authenticating the Databricks CLI (Windows/macOS/Linux)

# Install (Windows)
winget install Databricks.DatabricksCLI

# Or macOS/Linux (Homebrew)
brew tap databricks/tap && brew install databricks

# Login (OAuth, recommended)
databricks auth login --host https://adb-<workspace-id>.<region>.azuredatabricks.net

After login, databricks secrets ... commands are ready to use. (Databricks Documentation)

Giving others access to a scope

Scopes have ACLs—grant READ/MANAGE to groups, not individuals:

# Let data-engineers read secrets from this scope
databricks secrets put-acl my-scope data-engineers READ

(Or use the workspace UI to manage ACLs.) (Microsoft Learn)

Real-world usage patterns

1) JDBC without hard-coded creds

user = dbutils.secrets.get("jdbc","username")
pwd  = dbutils.secrets.get("jdbc","password")

df = (spark.read.format("jdbc")
      .option("url", "...")    # jdbc:sqlserver://...
      .option("dbtable","dbo.orders")
      .option("user", user)
      .option("password", pwd)
      .load())

(Databricks Documentation)

2) Mounting / accessing cloud storage

Keep client secrets or SAS tokens in a scope and pull them inside a job; avoid embedding keys into clusters or notebooks. (Same pattern as above.)

3) DLT & Jobs config

Keep env-specific endpoints and tokens as secrets; read them in pipeline notebooks. (dbutils.secrets.get(...) works in jobs and DLT just the same.) (Databricks Documentation)

Best practices (from the field)

Prefer AKV-backed for central rotation & audit; use Databricks-backed for local or short-lived needs. (Microsoft Learn)
Group-based ACLs only (READ/MANAGE); avoid per-user sprawl. (Microsoft Learn)
Never print secrets; they’re redacted, but don’t log them or pass to UDFs that might spill to disk. (Stack Overflow)
Separate scopes by role/app (for example, ingestion, bi, ml), not by person. (Databricks Documentation)
Automate with CLI or Terraform for repeatable environments. (Terraform Registry)

Troubleshooting quickies

“Scope created but can’t read” → Check ACLs; user/group needs READ on the scope. (Microsoft Learn)
AKV scope creation fails → Ensure Contributor/Owner on the Key Vault for the identity creating the scope. (kb.databricks.com)
No transcript/what was said in the video? → This blog expands the video’s topics with official doc flows and examples.

Wrap-up

Secret scopes let you keep credentials out of code and under governance. Start with a Databricks-backed scope to see the flow end to end, then move production credentials into Azure Key Vault-backed scopes and lock access with group ACLs. The APIs are simple, the operational posture is strong, and your notebooks/jobs stay clean and safe. (YouTube, Microsoft Learn, Databricks Documentation)

Tags: Databricks

Category:

Databricks