✅ What Is a Service Principal in Databricks?
A service principal in Databricks represents a non-human identity — like an application, automation tool, or CI/CD pipeline — used to securely access Databricks resources without using a personal user account.
🔧 Common Use Cases with Examples
| Use Case | Example | Why Use a Service Principal? |
|---|---|---|
| 🔁 Automated Jobs | A data pipeline in Azure Data Factory (ADF) triggering a Databricks job | To allow ADF to authenticate to Databricks securely without hardcoding credentials |
| 🚀 CI/CD Deployments | GitHub Actions / Azure DevOps pushing notebooks, jobs, clusters using REST APIs | To allow CI/CD tools to deploy infrastructure to Databricks securely and reproducibly |
| 📦 Data Ingestion | A Kafka consumer pushing real-time data into Delta Lake | So the ingestion tool (not a person) can write to Databricks securely |
| 📊 Power BI / Reporting Tools | Power BI connecting to a SQL Warehouse using OAuth via a service principal | For secure and auditable access to query data without personal credentials |
| 🔐 Unity Catalog Permissions | Assigning roles and access to tables, volumes, schemas | So that tools like Airflow, ADF, or APIs can read/write data under controlled permissions |
| 🛡️ Auditing & Security | Keeping audit logs clean by separating human vs automation identities | Improves security visibility and compliance tracking |
✅ How to Create and Use One (Summary)
- Register an app in Azure Active Directory.
- Create a secret or certificate.
- Assign that app as a service principal in Databricks via Admin Console.
- Grant it permissions on:
- Clusters
- Jobs
- Catalog objects (via Unity Catalog)
- Use the token/secret to authenticate in automation (e.g., via REST API).
🔐 Bonus: Sample Python Usage with Token
import requests
token = "Bearer <sp-access-token>"
workspace_url = "https://<your-workspace>.azuredatabricks.net"
response = requests.get(
f"{workspace_url}/api/2.0/clusters/list",
headers={"Authorization": token}
)
print(response.json())
