🧠 How to Run Your First Notebook in Databricks: From Beginner to Advanced

Whether you’re a data analyst, engineer, scientist, or just a curious tech enthusiast, Databricks offers a powerful platform to explore big data and AI effortlessly using notebooks. In this guide, we’ll walk you through the complete journey — from setting up your environment to mastering advanced notebook workflows in Databricks.

🔰 Table of Contents

1. What is a Databricks Notebook?
2. Step 1: Set Up Your Databricks Workspace
3. Step 2: Create a New Cluster
4. Step 3: Create and Open a Notebook
5. Step 4: Write Your First Code (Beginner)
6. Step 5: Attach Notebook to Cluster
7. Step 6: Execute Code Cells
8. Step 7: Use Magic Commands and Shortcuts
9. Step 8: Import Data into Your Notebook
10. Step 9: Visualize Data (Charts/Graphs)
11. Step 10: Schedule Notebooks with Jobs
12. Step 11: Version Control with Git
13. Step 12: Best Practices for Notebook Development
14. Final Thoughts

🔎 What is a Databricks Notebook?

A Databricks notebook is a collaborative, interactive environment where you can write code in Python, SQL, R, Scala, or Java, and view results immediately. It’s like Jupyter Notebook but powered by Apache Spark under the hood — perfect for data analytics and machine learning workflows.

✅ Step 1: Set Up Your Databricks Workspace

You can get started with either:

• Databricks Community Edition (Free): https://community.cloud.databricks.com
• Paid cloud workspaces on Azure or AWS

Create an Account:

• Sign up → Verify Email → Log in

⚙️ Step 2: Create a New Cluster

A cluster is a collection of computation resources (Spark jobs run here).

How to create:

• Go to Compute tab
• Click Create Cluster
• Choose:
  • Cluster name
  • Runtime version (e.g., 14.3 LTS for latest features)
  • Autoscaling & node type (select small for testing)

🔄 Wait for the cluster to be in Running state before attaching a notebook.

📓 Step 3: Create and Open a Notebook

• Go to Workspace
• Click Create → Notebook
• Name your notebook (e.g., MyFirstNotebook)
• Select Default Language: Python / SQL / Scala / R

✍️ Step 4: Write Your First Code (Beginner)

Try running a simple Python print statement:

print("Hello, Databricks!")

Or a simple SQL query:

%sql
SELECT "Hello from SQL" AS greeting;

🔗 Step 5: Attach Notebook to Cluster

• At the top of the notebook, click Detached → select your running cluster
• Now your code will execute on that cluster

▶️ Step 6: Execute Code Cells

• Press Shift + Enter or click the Run icon on each cell
• You can add multiple cells using the + sign
• Check output immediately below the cell

🧙 Step 7: Use Magic Commands and Shortcuts

Databricks supports magic commands to switch between languages or shell access:

Example:

%fs ls /databricks-datasets

Useful Shortcuts:

• Esc + A → Insert cell above
• Esc + B → Insert cell below
• Ctrl + / → Comment/Uncomment

📁 Step 8: Import Data into Your Notebook

Option 1: Upload file manually

• Click Data tab → Add Data → Upload File
• Access it using:

df = spark.read.csv("/FileStore/tables/mydata.csv", header=True)

Option 2: Use built-in datasets

df = spark.read.csv("/databricks-datasets/airlines/part-00000", header=True)

📊 Step 9: Visualize Data (Charts/Graphs)

• Run a DataFrame cell
• Click on the “+Visualization” icon
• Choose from:
  • Bar Chart
  • Line Chart
  • Pie Chart
  • Scatter Plot
  • Map

You can also use:

display(df.groupBy("Year").count())

⏰ Step 10: Schedule Notebooks with Jobs

1. Go to Workflows → Jobs
2. Click Create Job
3. Select:
   • Notebook path
   • Cluster
   • Schedule (e.g., Daily at 8 AM)
4. Monitor job history from dashboard

This is great for ETL or automated reports.

🧬 Step 11: Version Control with Git

Integrate your notebook with GitHub or Azure DevOps.

Steps:

1. Click on Revision History
2. Link to a Git provider
3. Clone repo and commit directly from the notebook interface

💡 Step 12: Best Practices for Notebook Development

🧾 Final Thoughts

Databricks notebooks combine the power of big data with the simplicity of notebooks — making them a go-to tool for everyone from analysts to AI engineers. Once you’ve mastered your first notebook, explore Delta Lake, MLflow, and Unity Catalog to expand your skills even further.

📌 Bonus: Sample Notebook Flow

🚀 Ready to launch your first project?

Try building a notebook that:

• Loads a dataset
• Cleans & filters it
• Displays visualizations
• Runs on a schedule

Happy Notebooks! 🧪

Mohammad Gufran Jahangir