,

Audit Logs Not Capturing Unity Catalog Actions in Databricks

Posted by

Introduction

Audit logs in Databricks Unity Catalog track data access, modifications, security changes, and governance actions. If audit logs are missing Unity Catalog actions, it could lead to compliance risks, security gaps, and lack of visibility into user activities.

🚨 Common issues with Unity Catalog audit logs:

  • No Unity Catalog events in audit logs.
  • Metadata changes (CREATE, ALTER, DROP) are not logged.
  • Data access logs (SELECT, INSERT, DELETE) are missing.
  • Databricks workspace logs general events but not Unity Catalog-specific actions.

This guide helps troubleshoot and enable full Unity Catalog audit logging in Databricks.


1. Verify That Audit Logging Is Enabled in Databricks

Symptoms:

  • No audit logs are captured for Unity Catalog events.
  • Audit logs are empty or only contain workspace events.

Causes:

  • Audit logging is not enabled in the Databricks workspace.
  • Logging is configured incorrectly (wrong storage account, missing permissions).

Fix:

Check if audit logging is enabled:

databricks account settings get-audit-log-config

Enable audit logging (Admin Required):

databricks account settings enable-audit-logs --storage-account <storage-name> --container <container-name>

For AWS, ensure CloudTrail is configured to capture Databricks events:

aws cloudtrail describe-trails

For Azure, ensure diagnostic settings include Unity Catalog logs:

  1. Go to Azure Portal → Monitor → Diagnostic Settings.
  2. Enable logs for Databricks and Unity Catalog.

2. Check If Unity Catalog Actions Are Included in Logging Configuration

Symptoms:

  • Only workspace-level events are captured, but Unity Catalog actions are missing.
  • No logs for catalog, schema, or table changes.

Causes:

  • The audit log does not include Unity Catalog operations.
  • Logging is capturing only cluster and job events.

Fix:

Ensure that Unity Catalog-specific logs are enabled:

databricks logging enable --log-type UNITY_CATALOG

Enable logging for metadata operations (CREATE, ALTER, DROP):

databricks audit-logs configure --include-metadata-changes true

Enable fine-grained logging for SQL queries:

databricks logging enable --log-type SQL_QUERIES

For AWS, add Unity Catalog logs to CloudTrail:

aws cloudtrail put-event-selectors --trail-name <trail-name> --event-selectors '[{"ReadWriteType": "All", "IncludeManagementEvents": true, "DataResources": [{"Type": "AWS::S3::Object", "Values": ["arn:aws:s3:::databricks-audit-logs/*"]}]}]'

3. Verify That Unity Catalog Logs Are Sent to Cloud Storage

Symptoms:

  • Logs exist in Databricks UI but are not stored in cloud storage.
  • Log files are missing in AWS S3, Azure ADLS, or GCS.

Causes:

  • Logging storage path is incorrect or not accessible.
  • IAM roles do not allow Databricks to write logs to cloud storage.

Fix:

Check storage configuration for audit logs:

databricks account settings get-audit-log-config

Ensure Databricks has permissions to write logs to S3/ADLS/GCS:

  • AWS IAM policy for S3 logging:
{
  "Effect": "Allow",
  "Action": ["s3:PutObject", "s3:GetObject"],
  "Resource": "arn:aws:s3:::databricks-audit-logs/*"
}
  • Azure ADLS permissions:
az role assignment create --assignee <databricks-service-principal> --role "Storage Blob Data Contributor" --scope /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage-name>

Verify logs are being written to storage:

aws s3 ls s3://databricks-audit-logs/
az storage blob list --container-name <container-name> --account-name <storage-name>

4. Check Unity Catalog Permission Auditing (GRANT, REVOKE)

Symptoms:

  • No logs for permission changes (GRANT, REVOKE, ALTER).
  • Security team cannot track access modifications.

Causes:

  • Permission change events are not enabled in audit logs.
  • Unity Catalog does not log permission changes by default unless configured.

Fix:

Enable logging for GRANT and REVOKE actions:

databricks logging enable --log-type PERMISSION_CHANGES

Manually verify permission changes using Unity Catalog SQL queries:

SHOW GRANTS ON CATALOG main;
SHOW GRANTS ON SCHEMA default;

For AWS, enable permission tracking in CloudTrail:

aws cloudtrail put-event-selectors --trail-name <trail-name> --event-selectors '[{"ReadWriteType": "All", "IncludeManagementEvents": true}]'

5. Ensure Query History Logging for Data Access Audits

Symptoms:

  • No logs for SELECT, INSERT, UPDATE, DELETE queries.
  • Cannot track who accessed specific tables.

Causes:

  • Query history logging is disabled.
  • Unity Catalog SQL queries are not being logged.

Fix:

Enable query logging:

databricks logging enable --log-type QUERY_HISTORY

Manually check query logs:

SHOW HISTORY ON TABLE my_catalog.my_schema.my_table;

For Databricks SQL warehouses, enable logging for queries:

  1. Go to SQL Warehouses → Edit → Enable Query Logging.

6. Troubleshooting Step-by-Step

Step 1: Verify That Audit Logs Are Enabled

databricks account settings get-audit-log-config

Step 2: Check If Unity Catalog Logging Is Included

databricks logging enable --log-type UNITY_CATALOG

Step 3: Ensure Logs Are Being Written to Storage

aws s3 ls s3://databricks-audit-logs/
az storage blob list --container-name <container-name> --account-name <storage-name>

Step 4: Verify Permissions for Writing Audit Logs

  • AWS: Ensure S3 IAM permissions allow Databricks to write logs.
  • Azure: Ensure Storage and Key Vault permissions are set correctly.

Step 5: Manually Check Query and Permission History

SHOW HISTORY ON TABLE my_catalog.my_schema.my_table;
SHOW GRANTS ON CATALOG main;

Best Practices for Capturing Unity Catalog Audit Logs

Ensure Audit Logging Is Enabled for Unity Catalog

databricks account settings enable-audit-logs --storage-account <storage-name>

Enable Logging for Metadata, Permissions, and Queries

databricks logging enable --log-type QUERY_HISTORY
databricks logging enable --log-type PERMISSION_CHANGES

Verify That Logs Are Being Stored in Cloud Storage

aws s3 ls s3://databricks-audit-logs/

Use CloudTrail (AWS) or Azure Monitor for Additional Logs

aws cloudtrail describe-trails

Conclusion

If Unity Catalog audit logs are missing or incomplete, check:
Audit logging is enabled in Databricks settings.
Unity Catalog-specific logs are turned on.
Logs are correctly stored in AWS S3, Azure ADLS, or Google Cloud Storage.
Permission changes and SQL queries are logged properly.

By following this guide, you can ensure that all Unity Catalog actions are audited for compliance, security, and governance.

guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x