Introduction
Access control in Databricks ensures secure collaboration, data governance, and compliance across teams. However, misconfigured permissions, missing role assignments, and authentication failures can lead to restricted access, unauthorized data exposure, or operational disruptions.
In this guide, we will explore common access control issues in Databricks, their root causes, and best practices to ensure a secure and well-managed Databricks environment.
Understanding Access Control in Databricks
Databricks supports multiple layers of access control:
- Workspace Access Control – Manages user roles and permissions at the workspace level.
- Cluster Access Control – Defines who can start, stop, or modify clusters.
- Table and Data Access (Unity Catalog, ACLs) – Controls access to databases, tables, and files.
- Job Access Control – Manages permissions for job execution.
- Secret Management Access – Restricts access to stored secrets and credentials.
🚨 Access control issues can cause:
- Permission Denied Errors for users.
- Unauthorized Access to restricted data.
- Job and Workflow Failures due to incorrect role assignments.
Common Access Control Issues in Databricks
1. User Cannot Access Databricks Workspace
Symptoms:
- Users receive 403 Forbidden or Unauthorized errors when logging in.
- Access requests remain pending approval for extended periods.
Causes:
- User is not assigned a Databricks workspace role.
- Misconfigured SCIM/SSO integration with Azure AD, Okta, or AWS IAM.
Fix:
- Ensure user has “Databricks Admin” or “User” role assigned in the workspace.
- Check if SCIM provisioning is properly set up for identity management.
- Review SSO authentication logs for failures in Okta/Azure AD.
📌 Example: Assigning a User Role in Databricks (Azure AD)
az ad group member add --group "Databricks Users" --member-id <user-object-id>
2. Users Unable to Start or Modify Clusters
Symptoms:
- Users cannot start/terminate clusters.
- Error: “You do not have permission to start/modify this cluster.”
Causes:
- Cluster access control settings prevent the user from managing clusters.
- User lacks “Can Restart” or “Can Attach To” permissions on the cluster.
Fix:
- Go to Clusters → Edit Permissions and assign the correct permissions.
- Ensure that users are part of the Databricks groups with cluster access.
📌 Example: Assigning Cluster-Level Permissions via API
{
"cluster_id": "1234-567890-abcd",
"access_control_list": [
{
"user_name": "user@example.com",
"permission_level": "CAN_RESTART"
}
]
}
3. Permission Denied on Tables or Databases (Unity Catalog)
Symptoms:
- Users cannot run SELECT, INSERT, or DELETE queries.
- Error: “User does not have USE CATALOG or SELECT permission.”
Causes:
- User lacks appropriate ACLs in Unity Catalog.
- Misconfigured privileges on databases, schemas, or tables.
Fix:
- Grant database-level and table-level privileges using SQL.
- Ensure users have
USE CATALOG
permission before accessing databases.
📌 Example: Granting Table Permissions in Unity Catalog
GRANT SELECT ON TABLE sales_data TO `user@example.com`;
GRANT USE CATALOG ON CATALOG analytics TO `user@example.com`;
4. Jobs Fail Due to Insufficient Permissions
Symptoms:
- Scheduled jobs fail with access-related errors.
- Error: “Insufficient privileges to run job.”
Causes:
- User or service principal lacks job execution permissions.
- Role assignments do not include “Can Manage” or “Can Run” privileges.
Fix:
- Assign job permissions via Jobs → Permissions in Databricks UI.
- Use service principals for executing production jobs.
📌 Example: Granting Job Execution Permission via API
{
"job_id": "6789-job-example",
"access_control_list": [
{
"group_name": "Data Engineers",
"permission_level": "CAN_MANAGE"
}
]
}
5. Access Denied to Secret Scopes
Symptoms:
- Users cannot retrieve stored secrets for database connections or API keys.
- Error: “Permission Denied: Secret scope access is restricted.”
Causes:
- User is not authorized to access the secret scope.
- Secret permissions are restricted to specific roles.
Fix:
- Assign secret scope permissions using the Databricks CLI.
- Ensure service principals have read access to necessary secrets.
📌 Example: Assigning Secret Access via Databricks CLI
databricks secrets put-acl --scope <scope-name> --principal user@example.com --permission READ
Step-by-Step Troubleshooting Guide
1. Verify User and Role Assignments
- Check if the user is assigned to the correct Databricks group.
- Ensure workspace admins have granted necessary permissions.
2. Debug Access Denied Errors in Logs
- Review Databricks Audit Logs (AWS CloudTrail, Azure Monitor).
- Check Unity Catalog permission logs using:
SHOW GRANTS ON TABLE sales_data;
3. Test Role-Based Access Control (RBAC) Configurations
- Run a test query using an affected user account.
- Validate permissions using:
SHOW GRANTS TO `user@example.com`;
4. Update Service Principal Permissions for Automated Workloads
- Ensure service principals have required compute and storage permissions.
- Assign least privilege access using IAM roles.
Best Practices for Managing Access Control in Databricks
✅ Use Unity Catalog for Centralized Access Control
- Define consistent access policies across all workspaces.
- Prevent privilege escalation by enforcing role-based access control (RBAC).
✅ Enable Fine-Grained Permissions on Clusters & Jobs
- Assign users to Databricks Groups rather than granting permissions individually.
- Restrict cluster modifications to admins and specific engineering teams.
✅ Use Service Principals for Secure Job Execution
- Prevent credential sharing by assigning dedicated service accounts for jobs.
- Store API keys securely in Databricks Secrets instead of notebooks.
✅ Regularly Audit Access Logs
- Set up automated alerts for unauthorized access attempts.
- Monitor Databricks audit logs to detect permission changes.
Real-World Example: Fixing a Table Access Issue in Unity Catalog
Scenario:
A data engineer was unable to query a Delta table stored in Unity Catalog, receiving a “Permission Denied” error.
Root Cause:
- The engineer lacked USE CATALOG permission on the database.
- The default permissions did not include their role.
Solution:
- Granted necessary permissions:
GRANT USE CATALOG ON CATALOG analytics TO `engineer@example.com`;
GRANT SELECT ON TABLE sales_data TO `engineer@example.com`;
2. Updated Databricks Groups:
- Added Data Engineers group with appropriate privileges.
📌 Impact:
- Resolved access issues instantly, ensuring smooth data queries.
- Improved access control visibility for admins.
Conclusion
Access control issues in Databricks arise due to misconfigured permissions, missing role assignments, or authentication failures. By leveraging Unity Catalog, setting fine-grained permissions, and proactively auditing access logs, organizations can ensure secure, efficient, and well-governed Databricks environments.