50 interview questions and answers for an Azure Databricks Platform Engineer

Here are 50 interview questions and answers for an Azure Databricks Platform Engineer, divided by skill level and covering key areas (core, advanced, and scenario-based):

🔹 Essential Level (Core – 1 to 20)

1. What is Azure Databricks?
   • A unified data analytics platform built on Apache Spark, optimized for Azure cloud for big data and AI workloads.
2. How is Azure Databricks different from HDInsight?
   • Databricks provides a collaborative workspace with ML and Spark support, whereas HDInsight is a broader Hadoop-based platform.
3. What is a cluster in Databricks?
   • A set of computation resources (driver and workers) used to run notebooks and jobs.
4. Difference between All-Purpose and Job clusters?
   • All-Purpose: Interactive development.
   • Job: For production jobs triggered via jobs API or pipeline.
5. What is a notebook in Databricks?
   • A web-based interface to write and execute code in multiple languages like Python, Scala, SQL, and R.
6. What is the role of dbutils?
   • Utilities to interact with DBFS, secrets, widgets, etc.
7. What is Unity Catalog?
   • Centralized governance for data access, lineage, audit, and discovery across workspaces.
8. Explain the Databricks file system (DBFS).
   • A distributed file system over object storage in Azure (ADLS/Blob).
9. What is a Delta Lake?
   • A storage layer enabling ACID transactions on top of Data Lakes.
10. What is a Spark job vs. stage vs. task?
    • Job = triggered action, Stage = execution plan, Task = execution unit.
11. What are widgets in Databricks?
    • Interactive input controls for notebooks (text, dropdowns, multiselect).
12. What is Auto-scaling in Databricks?
    • Databricks automatically adds/removes workers based on workload.
13. What are the Databricks runtimes (LTS, Photon)?
    • Pre-built environments with specific Spark + ML/DL + GPU + performance features.
14. Difference between Azure Data Lake and Delta Lake.
    • ADLS: Storage only.
    • Delta Lake: Storage + transactional layer + schema enforcement.
15. What is Lakehouse architecture?
    • Combines Data Warehouse (structured, reliable) with Data Lake (scalable, raw).
16. How is data versioned in Delta Lake?
    • Using transaction logs (_delta_log folder), supports time travel.
17. How do you secure notebooks in Databricks?
    • Workspace ACLs, Git integration, credential passthrough, and Unity Catalog RBAC.
18. What are cluster pools?
    • A pool of idle VMs used to reduce cluster start time.
19. What is a workspace in Databricks?
    • Logical container for notebooks, clusters, jobs, and libraries.
20. What is a secret scope in Databricks?
    • Secure place to store and access secrets (via Azure Key Vault or Databricks-managed).

🔹 Advanced Level (21 to 35)

21. How do you implement CI/CD with Databricks and Azure DevOps?
    • Git integration → Repos → YAML pipeline → dbx or CLI deployment → promote to higher envs.
22. What is Photon in Databricks?
    • Native vectorized execution engine that boosts SQL performance (C++).
23. How do you monitor and debug Spark jobs in Databricks?
    • Use Spark UI: Jobs tab, Stages, Executors, SQL, Thread Dump analysis.
24. What is the function of VACUUM in Delta Lake?
    • Physically deletes old files no longer referenced in Delta tables.
25. What is the difference between append, overwrite, and merge in Delta?
    • Append: Add data, Overwrite: Replace, Merge: UPSERT.
26. Explain data lineage in Unity Catalog.
    • Tracks data flow from source to transformations to output (tables, views, notebooks).
27. How do you enforce data governance in Unity Catalog?
    • Grant privileges on catalog/schema/table level; use RBAC, tags, audit logs.
28. What is the difference between Azure SQL Database and Synapse Analytics?
    • SQL DB: OLTP.
    • Synapse: OLAP with MPP for big data analytics.
29. How do you optimize cost in Databricks?
    • Auto-terminate idle clusters, use job clusters, enable Photon, review cluster sizing, use pools.
30. How does Delta Lake handle schema evolution?
    • Using mergeSchema or overwriteSchema options while writing.
31. Explain checkpointing in structured streaming.
    • Saves state to resume processing after failure.
32. How to troubleshoot skewed stages in Spark?
    • Check stage time in Spark UI, look for uneven tasks, optimize joins or repartition.
33. How to connect Databricks with Azure Data Factory?
    • Using Web Activity or Databricks notebook activity via token authentication.
34. Explain workspace-level vs. Unity Catalog-level security.
    • Workspace: ACL-based
    • UC: Catalog/schema/table level using RBAC.
35. Difference between Delta Sharing and traditional APIs.
    • Open protocol to securely share live data without copying.

🔹 Scenario-Based (36 to 50)

36. How would you set up a dev → test → prod pipeline for notebooks?
    • Use Git repos, environment-specific config, CI/CD YAML pipelines, and approval gates.
37. A Spark job is taking 2 hours. How do you debug it?
    • Use Spark UI → Job tab → Identify long stages → Check executor skew, GC time, or wide transformations.
38. Your Delta table queries are slow. What do you check?
    • Check Z-ordering, data size, partitions, and whether Delta caching is used.
39. How would you manage secrets in multiple environments?
    • Create environment-specific secret scopes, integrate with Azure Key Vault.
40. If a user sees a permission denied on a table, what do you do?
    • Check Unity Catalog grants using SHOW GRANTS or UI access control.
41. You have hundreds of notebooks in Dev. How do you promote only a few to Prod?
    • Use branch-based Git workflow or folder-specific release pipelines.
42. How do you automate cluster lifecycle for jobs?
    • Define job clusters in job JSON config or use cluster pools.
43. Explain a use case where you’d prefer a job cluster over an all-purpose one.
    • For scheduled nightly ETL jobs that don’t need interactivity.
44. What’s your backup strategy for Delta tables?
    • Use checkpointing and versioning (time travel); optionally export to ADLS.
45. How do you audit user activity in Databricks?
    • Use system.access.audit tables or Azure Monitor diagnostic logs.
46. A user mistakenly deletes a table. How do you recover?
    • Use Delta Lake time travel: VERSION AS OF or TIMESTAMP AS OF.
47. Explain the benefits of Z-Ordering in Delta Lake.
    • Optimizes file skipping for faster queries by co-locating related data.
48. You need to run ML jobs nightly. What’s your setup?
    • Schedule notebook using Job API on a job cluster with required libraries.
49. What happens when two jobs write to the same Delta table?
    • Delta uses optimistic concurrency; one job may fail if conflict occurs.
50. Your workspace has too many idle clusters. How do you clean it up?
    • Enforce cluster auto-termination, monitor via audit logs, tag clusters for ownership.