Creating a Job Compute (also called a Job Cluster) in Databricks allows you to define a dedicated compute environment that is spun up only when your job runs — and then terminated automatically. This is perfect for scheduled tasks like your group_user_mapping refresh.
Here’s a complete step-by-step guide to creating a Job Compute in Databricks 👇
✅ Step-by-Step: Create a Job Compute Cluster in Databricks
🧭 Step 1: Go to Jobs in the Sidebar
- In the Databricks UI, click on “Jobs & Pipelines” (left menu)
- Click “Create Job”
🧱 Step 2: Define Your Job
- Name: e.g.
Refresh_Group_User_Mapping - Click + Add task
- Task name:
refresh_table - Type: Notebook
- Source: Workspace
- Path: Select your notebook (e.g.,
User_Grouping) - Now under Compute, click:
+ New job cluster
⚙️ Step 3: Configure the Job Compute Cluster
You’ll now define your job cluster:
| Setting | Recommended | Notes |
|---|---|---|
| Cluster name | jobcluster-group-refresh | Optional (auto-generated if blank) |
| Databricks Runtime Version | Latest with LTS (e.g., 11.3 LTS) | Use Unity Catalog-compatible version |
| Node Type | Standard_DS3_v2 or default | Depends on your workload size |
| Worker count | Min: 1, Max: 2 (autoscale) | Or fix to 1 if small job |
| Enable autoscaling | ✅ | Saves cost on light jobs |
| Terminate after | 10–30 mins | Job clusters auto-terminate after job ends |
✅ Leave most advanced settings as default unless you need specific libraries.
🎯 Step 4: Save & Run
- After setting up the cluster, click Create Task
- Back in the Job overview, click Run Now to test it
- To schedule it, click “Add Schedule” (e.g., daily at 6 AM)
🧠 Why Use Job Compute?
| Feature | Benefit |
|---|---|
| 💸 Auto-termination | Saves cost — no idle billing |
| 🧼 Clean environment | Each run starts fresh |
| 📈 Scalable | Can scale up/down if autoscaling is enabled |
| 🔒 Isolated | No impact from shared users or other notebooks |
⚠️ Requirements
- You must be on a premium or enterprise Databricks workspace
- Unity Catalog jobs require a UC-compatible runtime
- You need permissions to create clusters (usually allowed by default)
🔄 Later: Modify Job Cluster Settings
If you want to update the job cluster later:
- Go to the job → click the task → click Edit compute
- Update instance size, autoscaling, etc.
