Introduction
Azure Data Lake Storage (ADLS) Gen2 is the preferred storage solution for Unity Catalog in Azure Databricks, providing secure, scalable, and high-performance data storage. However, configuration issues, missing permissions, or authentication problems can prevent ADLS Gen2 from working with Unity Catalog.
🚨 Common issues with ADLS Gen2 storage in Unity Catalog:
- Cannot assign ADLS Gen2 as the Unity Catalog metastore.
- “Access Denied” errors when trying to read/write data.
- ADLS Gen2 paths (
abfss://
) fail in SQL queries or notebooks. - Table creation or access fails in Unity Catalog.
This guide explores troubleshooting steps and fixes to successfully use ADLS Gen2 storage for Unity Catalog in Azure Databricks.
1. Verify That ADLS Gen2 Is Configured for Unity Catalog
Symptoms:
- Error: “Unity Catalog requires an Azure Data Lake Storage Gen2 container for the metastore.”
- Metastore assignment fails when specifying ADLS Gen2 as storage.
Causes:
- Unity Catalog requires an ADLS Gen2 storage account with hierarchical namespace (HNS) enabled.
- The specified ADLS Gen2 container is not configured correctly.
Fix:
✅ Ensure ADLS Gen2 is enabled with hierarchical namespace (HNS):
- Go to Azure Portal → Storage Accounts
- Select the storage account → Configuration
- Ensure “Hierarchical Namespace” is enabled.
✅ Assign the ADLS Gen2 storage account to Unity Catalog:
databricks unity-catalog metastores create --region eastus --storage-root abfss://<container>@<storage-account>.dfs.core.windows.net/
✅ Check if the metastore is properly assigned to ADLS Gen2:
SHOW METASTORES;
2. Ensure Correct Permissions for Unity Catalog on ADLS Gen2
Symptoms:
- Error: “Permission denied while accessing storage root.”
- Databricks notebooks or SQL queries fail when accessing ADLS Gen2 data.
Causes:
- Unity Catalog requires Storage Blob Data Contributor and Storage Account Contributor permissions on the ADLS Gen2 storage account.
- The Azure Managed Identity for Databricks does not have storage access.
Fix:
✅ Assign the required roles to the Databricks workspace identity:
az role assignment create --assignee <databricks-service-principal> --role "Storage Blob Data Contributor" --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage-name>
✅ Verify permissions using Azure CLI:
az role assignment list --assignee <databricks-service-principal>
✅ Grant access at the container level using Azure Storage Explorer:
- Open Azure Storage Explorer
- Navigate to Containers →
- Assign “Storage Blob Data Reader” and “Storage Blob Data Contributor” to your Databricks service principal.
✅ Verify Unity Catalog access using Databricks:
dbutils.fs.ls("abfss://<container>@<storage-account>.dfs.core.windows.net/")
3. Ensure Databricks Has the Correct Authentication for ADLS Gen2
Symptoms:
- Error: “AAD Token authentication failed.”
- Cannot authenticate Databricks workspace to ADLS Gen2.
Causes:
- Incorrect authentication method used in Databricks.
- Databricks Service Principal is not registered with ADLS Gen2.
Fix:
✅ Verify that the Databricks Service Principal is assigned to the ADLS Gen2 storage account:
az role assignment list --assignee <databricks-service-principal>
✅ Use Managed Identity authentication for Databricks in ADLS Gen2:
databricks unity-catalog metastores update --metastore-id <metastore-id> --storage-root abfss://<container>@<storage-account>.dfs.core.windows.net/
✅ For direct access, set up OAuth-based authentication in Databricks:
configs = {"fs.azure.account.auth.type": "OAuth",
"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
"fs.azure.account.oauth2.client.id": "<client-id>",
"fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope="my-secret-scope", key="client-secret"),
"fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<tenant-id>/oauth2/token"}
dbutils.fs.mount(
source="abfss://<container>@<storage-account>.dfs.core.windows.net/",
mount_point="/mnt/adls",
extra_configs=configs)
4. Validate ADLS Gen2 Storage Root in Unity Catalog
Symptoms:
- Error: “Invalid storage root for Unity Catalog.”
- Databricks jobs fail when writing to Unity Catalog tables.
Causes:
- Unity Catalog requires a specific storage format (
abfss://
) for ADLS Gen2. - The specified container path is incorrect.
Fix:
✅ Ensure the storage root is in the correct format:
abfss://<container>@<storage-account>.dfs.core.windows.net/
✅ Update the Unity Catalog metastore with the correct storage path:
databricks unity-catalog metastores update --metastore-id <metastore-id> --storage-root abfss://<container>@<storage-account>.dfs.core.windows.net/
✅ Verify storage root settings in Unity Catalog:
DESCRIBE METASTORE;
5. Troubleshooting Step-by-Step
Step 1: Verify ADLS Gen2 Configuration
az storage account show --name <storage-account> --query "properties.isHnsEnabled"
- If
"false"
, hierarchical namespace (HNS) is not enabled (recreate the storage account with HNS).
Step 2: Check Databricks Permissions on ADLS Gen2
az role assignment list --assignee <databricks-service-principal>
- Ensure Databricks has “Storage Blob Data Contributor” role on the ADLS Gen2 storage account.
Step 3: Test Direct Access to ADLS Gen2 from Databricks
dbutils.fs.ls("abfss://<container>@<storage-account>.dfs.core.windows.net/")
- If fails with “Access Denied”, fix IAM roles and permissions.
Step 4: Assign ADLS Gen2 to Unity Catalog
databricks unity-catalog metastores update --metastore-id <metastore-id> --storage-root abfss://<container>@<storage-account>.dfs.core.windows.net/
Step 5: Check Unity Catalog Storage Root Configuration
DESCRIBE METASTORE;
Best Practices for Using ADLS Gen2 with Unity Catalog
✅ Ensure ADLS Gen2 Has Hierarchical Namespace (HNS) Enabled
- Required for Unity Catalog to manage tables and metadata.
✅ Use the Correct Storage URL Format (abfss://
)
- Example:
abfss://<container>@<storage-account>.dfs.core.windows.net/
✅ Grant Databricks Service Principal the “Storage Blob Data Contributor” Role
- Assign permissions using:
az role assignment create --assignee <databricks-service-principal> --role "Storage Blob Data Contributor" --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage-name>
✅ Verify Unity Catalog Storage Root Assignments
- Run:
DESCRIBE METASTORE;
Conclusion
If you cannot use ADLS Gen2 storage for Unity Catalog, check:
✅ ADLS Gen2 is configured with Hierarchical Namespace (HNS) enabled.
✅ Correct IAM roles and storage permissions are set for Databricks.
✅ Authentication methods (Managed Identity, OAuth) are correctly configured.
✅ The storage root is in the correct format (abfss://…
).
Following this guide will ensure successful ADLS Gen2 integration with Unity Catalog in Azure Databricks.