,

Cannot Track Lineage for Unity Catalog Tables

Posted by

Introduction

Unity Catalog in Databricks provides automatic lineage tracking to visualize how datasets are used, transformed, and propagated across tables, notebooks, and workflows. However, lineage tracking might not work properly due to missing configurations, unsupported clusters, or restricted permissions.

🚨 Common Issues with Lineage Tracking in Unity Catalog:

  • Lineage UI is missing or does not show tables and views.
  • Lineage graph is incomplete or missing transformations.
  • Cannot see lineage data for Delta tables in Unity Catalog.
  • Lineage tracking is not updating after new queries or transformations.

This guide explores common causes of lineage tracking failures, troubleshooting steps, and best practices to enable proper lineage tracking in Unity Catalog.


1. Verify If Unity Catalog Lineage Is Enabled for Your Workspace

Symptoms:

  • Unity Catalog UI does not show the “Lineage” tab.
  • Lineage information is not recorded for new queries.

Causes:

  • Lineage tracking is not enabled for the workspace.
  • Databricks workspace is using an unsupported plan or region.

Fix:

Check if Unity Catalog Lineage is enabled in your workspace:

databricks workspace get-status
  • Ensure Unity Catalog is properly configured and assigned to a metastore.

If Unity Catalog is missing, enable it:

  • Go to Databricks Admin Console → Settings → Enable Unity Catalog.
  • Assign a Unity Catalog Metastore to your workspace.

Ensure your Databricks workspace is running on a supported cloud provider and region:

  • AWS: Lineage tracking is available in Databricks Premium and Enterprise editions.
  • Azure: Ensure your region supports Unity Catalog and lineage tracking.
  • GCP: Lineage tracking is available in Databricks Enterprise Edition.

2. Ensure You Are Using Unity Catalog Tables, Not Hive Metastore

Symptoms:

  • Lineage tracking is missing for tables and views.
  • Queries executed on Hive Metastore tables do not appear in lineage tracking.

Causes:

  • Lineage tracking only works for tables registered in Unity Catalog.
  • Hive Metastore tables are not tracked in Unity Catalog lineage.

Fix:

Verify if the table is stored in Unity Catalog:

SHOW CATALOGS;
SHOW TABLES IN my_catalog.my_schema;

If the table is in Hive Metastore, migrate it to Unity Catalog:

ALTER TABLE hive_metastore.default.my_table CONVERT TO DELTA;
CREATE TABLE my_catalog.my_schema.my_table AS SELECT * FROM hive_metastore.default.my_table;

Ensure you are querying the table using Unity Catalog schema:

SELECT * FROM my_catalog.my_schema.my_table;

3. Ensure You Are Using Unity Catalog-Compatible Clusters

Symptoms:

  • Lineage is not being recorded for queries executed in Databricks notebooks.
  • Lineage tab does not show any activity even after running queries.

Causes:

  • Legacy clusters do not support Unity Catalog lineage tracking.
  • Lineage tracking only works on Unity Catalog-enabled clusters.

Fix:

Check if your cluster supports Unity Catalog:

  • Go to Databricks UI → Clusters
  • Edit the cluster → Advanced Options → Ensure Unity Catalog is enabled

If necessary, create a new Unity Catalog-compatible cluster:

{
  "spark.databricks.cluster.profile": "singleNode",
  "spark.databricks.unityCatalog.enabled": "true"
}

Restart the cluster and re-run queries to verify lineage tracking.


4. Lineage Tracking Does Not Show Notebooks or Jobs

Symptoms:

  • Only tables appear in the lineage graph, but not notebooks or workflows.
  • Transformations executed in Databricks Jobs are missing from lineage.

Causes:

  • Lineage tracking requires Unity Catalog-compliant jobs and SQL warehouses.
  • Not all operations are automatically captured in lineage tracking.

Fix:

Ensure that the job is using Unity Catalog-compatible clusters.
If using Databricks Jobs, enable Unity Catalog for the job configuration.
Use SQL Warehouses for running transformations to capture full lineage.
Ensure that workflows and queries are executed within the Unity Catalog namespace.


5. Permissions Issues Prevent Lineage Tracking

Symptoms:

  • Lineage graph is missing tables, views, or users.
  • Users cannot view lineage information even though they have access to tables.

Causes:

  • User does not have “SELECT” or “USE CATALOG” permissions.
  • Lineage tracking is restricted to admins or specific roles.

Fix:

Grant the necessary permissions to users and roles:

GRANT USE CATALOG ON CATALOG my_catalog TO `user@example.com`;
GRANT SELECT ON TABLE my_catalog.my_schema.my_table TO `user@example.com`;

Ensure users have permission to view lineage tracking information.
If necessary, grant workspace-level access for Unity Catalog.


6. Lineage Graph Is Incomplete or Missing Transformations

Symptoms:

  • Some tables appear in lineage tracking, but others are missing.
  • Certain transformations (MERGE, DELETE, UPDATE) do not show up in lineage.

Causes:

  • Not all transformations are automatically tracked in Unity Catalog lineage.
  • Lineage updates can take time to process in Databricks UI.

Fix:

Check if lineage updates are delayed:

SHOW TABLE HISTORY my_catalog.my_schema.my_table;

Ensure that you are using Unity Catalog tables for all transformations.
For missing transformations, try using SQL Warehouses instead of interactive clusters.
Manually track transformations by logging metadata within notebooks.


7. Troubleshooting Steps to Fix Lineage Tracking Issues

Step 1: Check If Unity Catalog Lineage Is Enabled

databricks workspace get-status
  • If lineage tracking is not enabled, upgrade to a Premium or Enterprise workspace.

Step 2: Verify That You Are Using Unity Catalog Tables

SHOW CATALOGS;
SHOW TABLES IN my_catalog.my_schema;
  • If using Hive Metastore tables, migrate them to Unity Catalog.

Step 3: Ensure Your Cluster Supports Unity Catalog

  • Use Unity Catalog-compatible clusters for lineage tracking.

Step 4: Check If User Has Proper Permissions

GRANT USE CATALOG ON CATALOG my_catalog TO `user@example.com`;
  • Users must have the correct permissions to view lineage.

Step 5: Check If Notebooks and Workflows Are Capturing Lineage

  • Use SQL Warehouses instead of standard clusters for better lineage tracking.

Best Practices for Unity Catalog Lineage Tracking

Use Unity Catalog Tables Instead of Hive Metastore

  • Lineage tracking does not work for Hive Metastore tables.

Use SQL Warehouses Instead of Interactive Clusters

  • SQL Warehouses capture better lineage information for workflows.

Ensure Clusters Are Unity Catalog-Enabled

  • Legacy clusters cannot track lineage for Unity Catalog tables.

Assign Proper Permissions for Users to View Lineage

  • Ensure users and service accounts have the necessary permissions.

Manually Log Metadata for Complex Transformations

  • For complex MERGE, UPDATE, or DELETE operations, log metadata changes manually.

Conclusion

If lineage tracking is not working for Unity Catalog tables, check:
✅ Your Databricks workspace supports Unity Catalog.
You are using Unity Catalog tables, not Hive Metastore.
Clusters and SQL Warehouses are configured properly.
Users have the correct permissions to view lineage.
Databricks jobs and workflows are using Unity Catalog.

By following these steps, you can successfully enable lineage tracking for Unity Catalog and visualize how your data flows across your Databricks environment.

guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x