Introduction

The TestUC1004 – Invalid Token error in Databricks Unity Catalog indicates an authentication failure due to an expired, revoked, or improperly configured access token. This error is commonly encountered when accessing Unity Catalog, cloud storage (S3, ADLS, GCS), or during metastore setup. It can cause failures in API requests, data access, and table operations.

🚨 Common symptoms of TestUC1004 – Invalid Token:

• “TestUC1004 – Invalid Token” error during Unity Catalog operations.
• Access to cloud storage fails due to invalid credentials.
• API requests to Databricks REST API return 401 Unauthorized.
• Jobs fail intermittently due to expired tokens.

Common Causes and Fixes for TestUC1004

1. Expired or Revoked Access Token

Symptoms:

• Error: “Invalid token” when running Unity Catalog commands.
• API requests fail with a 401 Unauthorized response.
• Intermittent job failures due to token expiration.

Causes:

• The Databricks personal access token (PAT) has expired.
• The token was revoked or deleted by an admin.
• Token is hardcoded in scripts and not rotated regularly.

Fix:
✅ Check the expiration date of your token:

• Go to Databricks UI → User Settings → Access Tokens.
• Verify if the token is active and not expired.

✅ Generate a new access token:

databricks tokens create --lifetime-seconds 3600 --comment "New Unity Catalog Token"

✅ Update scripts or jobs to use the new token:

export DATABRICKS_TOKEN=<new-token>

✅ Avoid hardcoding tokens in scripts. Use environment variables or secret management tools.

2. Missing Azure AD Token or Expired Token (Azure Databricks)

Symptoms:

• Error: “Invalid token while accessing Azure Data Lake or Key Vault.”
• Cannot retrieve secrets from Azure Key Vault.
• Access to ADLS Gen2 fails intermittently.

Causes:

• Managed identity or service principal token has expired.
• Azure Active Directory (AAD) token was not refreshed.

Fix:
✅ Use Databricks CLI to check token status:

databricks configure --token

✅ If using a service principal, ensure token refresh logic is implemented:

az account get-access-token --resource https://datalake.azure.net/

✅ Ensure the Databricks cluster is configured to use Managed Identity:

{
  "spark.databricks.passthrough.enabled": "true"
}

✅ Assign the necessary Azure roles to the service principal:

az role assignment create --assignee <service-principal-id> --role "Storage Blob Data Contributor"

3. Invalid Token for AWS STS or S3 Access

Symptoms:

• Error: “Invalid token while accessing AWS S3 buckets.”
• Temporary security credentials fail in Databricks jobs.
• Data ingestion from S3 fails intermittently.

Causes:

• Temporary AWS credentials have expired.
• The IAM role is not properly configured for token-based access.

Fix:
✅ Ensure the AWS token is valid and has not expired:

aws sts get-session-token

✅ Use AWS AssumeRole for generating temporary tokens:

aws sts assume-role --role-arn <role-arn> --role-session-name "databricks-session"

✅ Attach the temporary token to the Databricks cluster configuration:

{
  "fs.s3a.access.key": "<access-key>",
  "fs.s3a.secret.key": "<secret-key>",
  "fs.s3a.session.token": "<session-token>"
}

4. Incorrect Cluster Configuration for Unity Catalog

Symptoms:

• Error: “Invalid token while accessing Unity Catalog.”
• Cannot run USE CATALOG or SHOW TABLES commands.
• Jobs fail due to token authentication issues.

Causes:

• Clusters are not configured to use Unity Catalog.
• Legacy clusters are incompatible with Unity Catalog.

Fix:
✅ Enable Unity Catalog support on clusters:

1. Go to Databricks UI → Clusters → Edit Cluster.
2. Enable Unity Catalog support in the Advanced options.
3. Restart the cluster.

✅ For SQL Warehouses, ensure Unity Catalog is enabled:

• Go to Databricks UI → SQL Warehouses → Edit → Advanced Settings.

Step-by-Step Troubleshooting Guide

1. Verify Token Status and Expiration

databricks tokens list

• Check if the token is active and not expired.

2. Check Cluster Configuration for Unity Catalog

Ensure that the cluster supports Unity Catalog and that token passthrough is enabled.

3. Test Cloud Storage Access with the Current Token

aws s3 ls s3://<bucket-name> --profile <your-profile>
az storage blob list --container-name <container-name>

4. Refresh or Generate a New Token if Necessary

az account get-access-token --resource https://datalake.azure.net/

Best Practices to Avoid TestUC1004 – Invalid Token Error

✅ Use Short-Lived Tokens and Rotate Regularly

• Avoid long-lived tokens; rotate tokens frequently.

✅ Implement Token Refresh Logic for Azure and AWS

• Use automatic token refresh for service principals and managed identities.

✅ Monitor Token Expiration and Access Logs

• Regularly monitor token expiration and failures in Databricks logs.

✅ Leverage Secret Management for Tokens

• Use Databricks Secrets or Azure Key Vault for managing tokens securely.

Conclusion

The TestUC1004 – Invalid Token error in Databricks Unity Catalog is typically caused by expired tokens, revoked credentials, or cluster misconfigurations. By verifying token status, enabling token passthrough, and implementing proper token refresh mechanisms, you can resolve this error and maintain secure, uninterrupted access to Unity Catalog and cloud services.

Mohammad Gufran Jahangir