Introduction
Secret management in Databricks allows users to securely store and retrieve sensitive information like API keys, database credentials, and cloud storage access keys. However, misconfigured secret scopes, missing permissions, or incorrect access methods can lead to authentication failures, job failures, and security risks.
🚨 Common issues caused by secret management problems in Databricks:
- Secrets not found or missing in workspace notebooks or jobs.
- AccessDenied errors when retrieving secrets.
- Databricks CLI fails to list, create, or access secrets.
- Secrets exposed in logs due to improper handling.
This guide explores common secret management problems, troubleshooting steps, and best practices to ensure secure and smooth secret usage in Databricks.
Understanding Secret Management in Databricks
Databricks provides two types of secret scopes:
- Databricks-backed secret scopes – Managed within Databricks.
- Azure Key Vault-backed secret scopes – Integrates with Azure Key Vault for external security.
💡 Secrets are stored in secret scopes and accessed using dbutils.secrets.get()
.
Example: Storing and Accessing Secrets in Databricks
# Create a secret scope
databricks secrets create-scope my-secret-scope
# Store a secret
databricks secrets put --scope my-secret-scope --key my-key
# Access secret in a notebook
dbutils.secrets.get(scope="my-secret-scope", key="my-key")
Common Secret Management Issues and Fixes
1. Secret Not Found (KeyError or NotAuthorized Error)
Symptoms:
- Error: “Secret not found: my-key”
- Error: “PERMISSION_DENIED: Cannot access scope: my-secret-scope”
- Secret scope exists but the specific secret is missing.
Causes:
- The secret scope was created but does not contain the expected key.
- The user lacks permission to access the secret scope.
- The Databricks CLI or API is querying the wrong secret scope.
Fix:
✅ Verify that the secret exists using Databricks CLI:
databricks secrets list --scope my-secret-scope
✅ Check user permissions for the secret scope:
databricks secrets list-acls --scope my-secret-scope
✅ Grant permission if missing:
databricks secrets put-acl --scope my-secret-scope --principal <user-email> --permission READ
✅ If using Azure Key Vault-backed secrets, check Key Vault access permissions.
az keyvault secret list --vault-name my-keyvault
2. Secrets Work in Notebooks But Fail in Jobs or CLIs
Symptoms:
- Notebook execution succeeds, but Databricks jobs fail with “Secret not found” error.
- Databricks CLI fails to access the secret scope.
- Permissions appear correct, but jobs still fail.
Causes:
- Databricks jobs run under different service principals that lack access.
- Cluster permissions do not allow job access to secrets.
- Interactive notebooks use different authentication than jobs.
Fix:
✅ Ensure the Databricks job or cluster has permissions to access the secret scope:
databricks secrets put-acl --scope my-secret-scope --principal <service-principal> --permission READ
✅ If using a job cluster, check that the cluster configuration allows access to secrets:
{
"spark.databricks.secret.enabled": "true"
}
✅ For Azure Key Vault-backed secrets, assign Managed Identity permissions to Key Vault:
az role assignment create --assignee <service-principal> --role "Key Vault Secrets User" --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<vault-name>
3. Secret Values Appear in Logs (Security Risk)
Symptoms:
- Sensitive credentials appear in Databricks logs.
- Secrets are accidentally printed in stdout or logs.
Causes:
- Using
print(dbutils.secrets.get())
, which exposes secrets in logs. - Passing secrets directly in SQL queries instead of using parameterized connections.
- Logging entire execution context, capturing secrets.
Fix:
✅ Never print secrets directly:
❌ Bad (Exposes Secrets)
print(dbutils.secrets.get("my-secret-scope", "my-key"))
✅ Good (Keeps Secrets Secure)
password = dbutils.secrets.get("my-secret-scope", "db-password")
✅ Use placeholders instead of injecting secrets in queries:
conn = f"jdbc:mysql://db-host:3306/db?user=myuser&password={{PASSWORD}}"
conn = conn.replace("{{PASSWORD}}", dbutils.secrets.get("my-secret-scope", "db-password"))
✅ For logging, redact sensitive information:
import logging
password = dbutils.secrets.get("my-secret-scope", "db-password")
logging.info(f"Database connection established. User: myuser, Password: [REDACTED]")
4. Azure Key Vault-Backed Secrets Failing in Databricks
Symptoms:
- Error: “Secret not found in Azure Key Vault.”
- Secrets work in the CLI but fail in Databricks notebooks or jobs.
- Intermittent authentication failures to Key Vault.
Causes:
- Databricks workspace is not linked to the correct Azure Key Vault.
- Key Vault access policies do not allow the Databricks service principal.
- Secrets were not properly synced to Databricks.
Fix:
✅ Verify Key Vault secrets using Azure CLI:
az keyvault secret list --vault-name my-keyvault
✅ Ensure Databricks has access to Azure Key Vault:
az keyvault set-policy --name my-keyvault --spn <databricks-service-principal> --secret-permissions get list
✅ Ensure the secret scope is correctly set to use Key Vault:
databricks secrets create-scope --scope my-kv-scope --scope-backend-type AZURE_KEYVAULT --resource-id <vault-resource-id>
5. Secret Scope Not Found in Databricks
Symptoms:
- Error: “Scope not found: my-secret-scope”
- Secret scope exists in another workspace but is missing in the current one.
- Databricks CLI fails to list the scope.
Causes:
- The secret scope was created in a different Databricks workspace.
- Cluster does not have permission to use secrets.
- Databricks CLI is pointing to the wrong workspace.
Fix:
✅ List available secret scopes to verify existence:
databricks secrets list-scopes
✅ Ensure the correct workspace is configured in Databricks CLI:
databricks configure --token
✅ Manually re-create the secret scope if necessary:
databricks secrets create-scope --scope my-secret-scope
Best Practices for Secure Secret Management in Databricks
✅ Use Databricks Secrets Instead of Hardcoding Credentials
- Never store passwords, API keys, or tokens in notebooks.
✅ Limit Secret Access to Only Required Users and Services
databricks secrets put-acl --scope my-secret-scope --principal "admins" --permission READ
✅ Use Azure Key Vault for Centralized Secret Management
- Avoid storing long-term secrets inside Databricks; use Azure Key Vault-backed secret scopes instead.
✅ Never Print Secrets in Logs
- Avoid logging secret values, use secure placeholder handling.
✅ Regularly Rotate Secrets
- Rotate database passwords, API keys, and cloud credentials regularly.
Conclusion
Secret management issues in Databricks can lead to authentication failures, security risks, and broken jobs. By ensuring correct permissions, using secure secret retrieval, and avoiding logging sensitive data, teams can safely store and use secrets in Databricks.