Databricks offers a multi-workspace architecture that can span across multiple cloud providers like AWS, Azure, and GCP. Managing this setup effectively requires understanding accounts, workspaces, user management, and metastore governance.
1. Databricks Account – The Control Plane
At the top level, you have the Databricks Account, which serves as the administrative control layer for:
- Managing workspaces across different environments (DEV, UAT, PROD, Pre-Prod).
- Centralizing user and group management.
- Setting up the Unity Catalog metastore for data governance.
The account exists independent of cloud provider and supports:
- AWS
- Azure
- Google Cloud Platform (GCP)
2. Workspaces – Your Data Projects’ Execution Environments
A workspace is an isolated environment where teams work on:
- Data ingestion
- Analytics
- Machine learning
- BI dashboards
Typical environment separation:
- DEV → For development and testing.
- UAT → For user acceptance testing.
- PROD → For live production workloads.
- Pre-Prod → For staging before production release.
Each workspace has a unique Workspace ID and URL in the format:
https://adb-<workspace_id>.<cloud>.databricks.com
For Azure, it might look like:
https://adb-1563232110.azuredatabricks.net
3. Unity Catalog Metastore – Central Governance Layer
The metastore in Unity Catalog allows:
- Centralized governance across all workspaces.
- Defining users, groups, and service principals (SP) once and applying them across environments.
- Assigning data permissions in a consistent and compliant manner.
This ensures that the same data access rules apply across DEV, UAT, PROD, and Pre-Prod without repetitive configuration.
4. Access & Security
- Users – Individual accounts for team members.
- Groups – Role-based collections of users.
- Service Principals (SP) – For automated workloads or integration services.
- Access is controlled both at workspace level and catalog/schema/table level.
5. Why This Structure Matters
A well-structured account and workspace setup:
- Simplifies data governance.
- Enables separation of environments for security and testing.
- Improves compliance by applying consistent rules across the organization.
- Makes it easier to manage multi-cloud setups.