,

Error While Configuring Metastore in Databricks Unity Catalog

Posted by

Introduction

Configuring a Unity Catalog Metastore in Databricks may fail due to misconfigured IAM roles (AWS), missing Azure AD permissions, incorrect storage setup, or workspace assignment errors. If you’re experiencing issues while configuring the Metastore, you might encounter errors like:

  • “Error: Could not create or assign Metastore.”
  • “Permission denied: Insufficient privileges to create Metastore.”
  • “Metastore already exists, but workspace cannot connect.”
  • “Cannot assign Metastore to workspace due to permission issues.”

🚨 Common issues when configuring a Unity Catalog Metastore:

  • Missing IAM permissions (AWS) or Key Vault permissions (Azure).
  • Databricks workspace is not properly assigned to the Metastore.
  • Cloud storage (S3, ADLS, GCS) is not correctly configured.
  • Conflicts with an existing Hive Metastore setup.

This guide provides step-by-step troubleshooting and solutions to resolve Metastore configuration errors in Databricks Unity Catalog.


1. Verify Databricks Plan and Region Support for Unity Catalog

Symptoms:

  • Unity Catalog UI is missing.
  • Metastore creation fails with a region-based error.
  • Error: “Unity Catalog is not available in this workspace.”

Causes:

  • Unity Catalog requires Databricks Premium or Enterprise edition.
  • Some cloud regions do not support Unity Catalog yet.

Fix:

Check Databricks plan:

  • Go to Admin Console → Settings → Workspace Settings
  • Ensure you are on Premium or Enterprise Edition (Standard does not support Unity Catalog).

Check Unity Catalog availability in your region:

  • AWS: Unity Catalog is supported in all Databricks-supported AWS regions.
  • Azure: Check supported regions here.
  • GCP: Unity Catalog is available for Enterprise Databricks on Google Cloud.

Upgrade your workspace to Premium if necessary.


2. Ensure IAM Permissions Are Set Correctly (AWS)

Symptoms:

  • Error: “IAM role does not have the required permissions to create a Metastore.”
  • Error: “S3 access denied when configuring Metastore.”

Causes:

  • The IAM role assigned to Databricks does not have access to Glue, S3, or necessary services.
  • S3 bucket policy is too restrictive to allow Metastore operations.

Fix:

Ensure IAM Role for Unity Catalog Has Required Permissions:

  • Attach the following AWS IAM policy to the Databricks role:
{
  "Effect": "Allow",
  "Action": [
    "glue:GetDatabase",
    "glue:CreateDatabase",
    "glue:UpdateDatabase",
    "glue:GetTables",
    "glue:CreateTable",
    "glue:UpdateTable",
    "glue:GetPartitions",
    "glue:CreatePartition",
    "s3:ListBucket",
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject"
  ],
  "Resource": "*"
}

Attach the IAM Role to Databricks:

aws iam attach-role-policy --role-name <your-role-name> --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

Verify IAM Role Assignment:

aws iam list-attached-role-policies --role-name <your-role-name>

Ensure S3 bucket has the correct policy to allow access:

{
  "Effect": "Allow",
  "Principal": {
    "AWS": "<your-databricks-iam-role-arn>"
  },
  "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::your-bucket-name/*"
}

Reattempt Metastore Creation:

databricks unity-catalog metastores create --region us-east-1 --s3-bucket my-bucket-name

3. Ensure Azure Key Vault and Storage Permissions Are Set Correctly (Azure)

Symptoms:

  • Error: “Access Denied: Key Vault secret not found.”
  • Error: “Storage account permission denied when assigning Metastore.”

Causes:

  • Databricks does not have proper Azure Key Vault permissions.
  • Storage account permissions do not allow Unity Catalog to store metadata.

Fix:

Grant Databricks Service Principal Access to Key Vault:

az keyvault set-policy --name <keyvault-name> --spn <databricks-service-principal> --secret-permissions get list set delete

Assign Storage Permissions to Databricks:

az role assignment create --assignee <service-principal> --role "Storage Blob Data Contributor" --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage-name>

Ensure Unity Catalog Metastore Can Access Storage:

databricks unity-catalog metastores create --region eastus --resource-group my-rg --storage-account my-storage

4. Assign the Metastore to the Databricks Workspace

Symptoms:

  • Error: “Cannot assign Metastore to workspace.”
  • Error: “Permission denied while assigning Metastore.”

Causes:

  • The Metastore was created but is not linked to the workspace.
  • Admin permissions are required to attach the Metastore.

Fix:

Assign the Metastore to the Workspace:

databricks unity-catalog metastores assign --metastore-id <metastore-id> --workspace-id <workspace-id>

Verify Assignment:

SHOW METASTORES;

Ensure Admin Permissions Are Assigned for Unity Catalog:

databricks permissions update --workspace-level --json '{"group_name": "admins", "permission_level": "CAN_MANAGE"}'

5. Resolve Conflicts With Existing Hive Metastore

Symptoms:

  • Error: “Conflicting Hive Metastore detected.”
  • Error: “Hive Metastore tables are not accessible via Unity Catalog.”

Causes:

  • Hive Metastore and Unity Catalog are separate metadata stores.
  • Unity Catalog requires migration of Hive tables.

Fix:

Convert Hive Metastore tables to Unity Catalog:

ALTER TABLE hive_metastore.default.my_table CONVERT TO DELTA;

Ensure New Tables Are Created in Unity Catalog Instead of Hive:

CREATE TABLE my_catalog.my_schema.new_table (id INT, name STRING);

6. Verify Cluster and SQL Warehouse Configurations

Symptoms:

  • Clusters do not display Unity Catalog tables.
  • Queries fail with catalog-related errors.

Fix:

Ensure Clusters Support Unity Catalog:

  1. Go to Databricks UI → Clusters
  2. Edit the cluster → Advanced options → Enable Unity Catalog

For SQL Warehouses, Enable Unity Catalog:

  1. Go to Databricks UI → SQL Warehouses
  2. Edit the warehouse settings to support Unity Catalog.

Restart the cluster after enabling Unity Catalog.


7. Step-by-Step Troubleshooting Guide

Step 1: Check If Unity Catalog Is Available

databricks workspace get-status

Step 2: Verify If a Metastore Exists

SHOW METASTORES;

Step 3: Test IAM and Storage Permissions

  • AWS: Ensure IAM policies allow Glue and S3 access.
  • Azure: Ensure Key Vault and Storage roles are correctly assigned.

Step 4: Assign Metastore to Workspace

databricks unity-catalog metastores assign --workspace-id <workspace-id>

Conclusion

If you cannot configure a Metastore for Unity Catalog, ensure that:
✅ Your Databricks workspace supports Unity Catalog (Premium or Enterprise).
✅ A Metastore exists and is assigned to the workspace.
IAM and cloud storage permissions are correctly configured.
Clusters and SQL Warehouses support Unity Catalog.

guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x