Week 13 - Big Data on Databricks
In Workspace & Unity Catalog you found hyf.nyc_yellow and ran SQL on a warehouse. This chapter is the other common path on Databricks: a notebook (interactive Python or SQL cells under Workspace) attached to a cluster, writing PySpark (the Python API for distributed DataFrame work). PySpark runs on Apache Spark (the open-source engine on that cluster).
By the end of this chapter, you should be able to:
show, count, write; not raw collect)A notebook is an interactive document of cells: you type code in a cell, run that cell only, and the result appears under it. Variables stay in memory for the next cell. On Databricks, a cell can run Python (including PySpark) or SQL. You pick the language per cell (or start a cell with %sql). You will try both: first a SQL cell on the warehouse you already know, then Python on a cluster. Try the cell idea before the theory:
https://lasse.be/simple-hyf-teach-widget/notebook-cells.html
Run the three cells in order (cell 3 reuses trips from cell 2). First load can take a moment while the browser starts Python. That is the interactive loop you will use on Databricks next.
So far this track you have mostly run Python as a script: a .py file that starts at the top, runs to the end, and exits. A notebook is different on purpose: like a long-lived Python session, the same idea as a REPL (Read-Eval-Print Loop: type a bit of code, run it, see the result, keep the variables and continue). Load raw_trips once, then try filters and aggregations many times without reloading.
Regular Python (.py script) |
Notebook | |
|---|---|---|
| How you run it | Top to bottom in one go | One cell at a time (any order you click) |
| After it finishes | Process exits; memory is cleared | Session stays up; variables stay in memory |
| Change one line | Usually re-run the whole file | Re-run that cell (and any cells that depend on it) |
| Output | Terminal print / logs | Result under the cell you ran |
| Good for | Fixed pipelines, apps, jobs | Exploration, demos, trying filters many times |
| Stale-state risk | Low (each run starts fresh) | High if you edit a cell and forget to re-run it |
That cell model comes from Jupyter Notebooks (files ending in .ipynb). Databricks notebooks use the same idea in the browser. You do not install Jupyter for class work this week; you create a Databricks notebook under Workspace.
Notebooks shine at interactive exploration. They are a weak place to ship a long-running product. On Databricks, many data scientists prefer notebooks for exploration; as a data engineer, you will often inherit them and later turn working notebooks into reliable Python (modules, jobs, schedules). This chapter teaches you to run notebooks safely. Migrating them into applications (and proving transforms with local pytest fixtures) is optional depth on Going Further.
<aside>
⚠️ Cell order is the order you ran, not the order on screen. The notebook remembers variables from every cell you executed, even if those cells sit below the one you are reading. Example: cell 1 sets trips = 3, cell 2 prints trips * 2. You run both and see 6. Later you change cell 1 to trips = 10 but only re-run cell 2. The print still uses the old value. Before you trust a result, run top to bottom (or clear state and re-run).
</aside>
Create one: open Workspace, right-click a folder (or use + New), choose Notebook. Set the default language to Python. You should see an empty cell ready for code, with a compute dropdown at the top of the page.
Sidebar + New menu with Notebook selected to create a notebook
New Python notebook: empty cell and compute dropdown in the toolbar
Leave the notebook open. Cells will not run until you attach compute in the next sections. For SQL cells that can be the warehouse; for Python it must be a cluster. Once a cluster is attached, Databricks defines spark for you.
<aside> ⌨️ Hands on: Create a new Python notebook under Workspace. Look at the empty cell and the compute dropdown at the top. Do not run cells yet.
</aside>
Nothing in a notebook runs without machines. Databricks gives you two kinds of compute this week:
hyf-dbt-warehouse for the count(*) in the last chapter. You can also attach a notebook to it and run SQL cells. It does not run Python or PySpark cells.count(), show(5), an aggregated table). It bills for every minute it is up, including time you spend reading between cells, so it auto-terminates when left idle.flowchart TB
subgraph cluster["cluster"]
drv["driver<br/>one machine"] -->|sends tasks| ex["executors<br/>one or more machines"]
ex -->|return partial results| drv
end
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class drv,ex box;
A notebook is only a document of cells. It does not have its own CPU or memory. Attach means: connect this notebook to compute so each cell has somewhere to run.
<aside> 🤓 Curious Geek: Who else is in the Spark cluster?
The diagram shows the two roles you use in class: a driver (plans and coordinates) and executors (do the work). In full Spark deployments a cluster manager (YARN, Kubernetes, or Databricks Compute) allocates those machines. Optional depth: Spark cluster architecture.
</aside>
At the top of the notebook, open the compute dropdown and attach hyf-dbt-warehouse (under serverless warehouses).
Attach to an existing compute resource: SQL Warehouse selected, hyf-dbt-warehouse, Attach
Set the cell language to SQL, or type %sql on the first line, then run:
SELECT count(*) AS trip_count
FROM hyf.nyc_yellow.raw_trips
Databricks notebook SQL cell: count(*) on raw_trips returning 128202548
You should see the same order of magnitude as in Catalog Explorer / the SQL editor (~128 million). That proves the notebook can run SQL when it is attached to the warehouse.
<aside>
⌨️ Hands on: Attach hyf-dbt-warehouse to your notebook. Run the count(*) SQL cell above. Confirm you get a large number on the order of 128 million.
</aside>
SQL on the warehouse worked. Python will not: the warehouse has no driver-plus-executors Spark session for PySpark cells. For the rest of this chapter you need the shared class cluster hyf-class-cluster.
Open the notebook compute dropdown again and attach hyf-class-cluster under General compute (not the warehouse). If the list shows another cluster (for example a teacher test cluster), ignore it: class PySpark work uses hyf-class-cluster. You do not need to open the Compute page for class work. Creating and inspecting clusters there is optional depth on Going Further.
Attach to an existing compute resource: General compute selected with hyf-class-cluster
If the cluster shows Terminated in the dropdown, start it from there (or ask your teacher if it is already warm). On the class subscription a cold start can take ten minutes or more and occasionally fails with a capacity error; details are in Gotchas.
When the status shows Connected (or similar), add a Python cell (hover between cells and click Code, or press b below a selected cell) and run:
Empty Python notebook cell ready for code
print("hello from the cluster")
Databricks notebook Python cell: print hello from the cluster with matching output
The first run can be slow if the cluster just woke up. If Python cells fail with messages about missing Spark or an unavailable session, the notebook is not attached to a cluster.
flowchart LR
A["1. Create<br/>notebook"] --> B["2. SQL cell<br/>on warehouse"]
B --> C["3. Attach<br/>cluster + Python cell"]
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class A,B,C box;
<aside>
⌨️ Hands on: Attach hyf-class-cluster to your notebook, and run print("hello from the cluster").
</aside>
Why not stay on the warehouse for everything? Capability first: the warehouse runs SQL (editor, dbt, and SQL notebook cells). Python and PySpark need a cluster. Cost second: the warehouse wins for short SQL (idle scales to zero); a warm cluster wins for interactive PySpark once it is running, but idle minutes still cost money until auto-termination. When you finish a hands-on, let the cluster auto-terminate (usually about 20 minutes of idle time on the class cluster). Treat that as a backup, not a reason to leave the cluster up while you take a long break.
<aside> 💡 The notebook compute dropdown may also list Serverless. That is not the SQL warehouse: it is Databricks-managed Spark for notebooks (docs). The required path this week is still the shared classic cluster so you see driver plus executors. Optional depth on Going Further.
</aside>
The lakehouse idea made the case for distributed computing: when data outgrows one machine, you spread the work. Apache Spark is the concrete open-source engine that does that. Databricks was founded by Spark's creators and hosts Spark (and more) as a managed platform. Keep four names separate:
| Name | What it is | What you do with it this week |
|---|---|---|
| Python | The language in the notebook cell | print(...), imports, control flow |
| Apache Spark | The distributed engine (open source) | Understand it: driver + executors run your job |
| PySpark | The Python API for Spark | Write notebook cells: spark.read, filter, groupBy |
| Databricks | The managed platform around Spark | Use the workspace, Unity Catalog, Delta tables, and hyf-class-cluster |
Plain Python runs in one process. PySpark looks like Python DataFrame code, but the heavy work runs across the cluster. Your notebook is a thin client: Python cells call PySpark; PySpark asks Spark to plan and run the job on the same driver and executors you attached above. You do not install Spark on your laptop for class.
flowchart TB
subgraph dbx["Databricks platform"]
subgraph spark["Apache Spark engine"]
pys["PySpark<br/>Python API you write"]
end
extras["Also: SQL warehouse, Jobs,<br/>Unity Catalog, notebooks, …"]
end
you["Your notebook cell"] --> pys
pys -->|"plans and runs the job"| spark
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class you,pys,extras box;
You write PySpark. Spark does the distributed work on the cluster. Databricks is the workspace that hosts Spark (and more than Spark).
For day-to-day work you are usually a consumer of a shared cluster: attach it, run cells, keep results small. For a 56K-row mart on one Postgres VM, Spark's startup overhead usually makes the job slower. This week's ~128 million trips are large enough that we run them on Databricks with Spark so you practice the multi-developer lakehouse path.
<aside> 💭 Single-machine engines such as DuckDB and Polars can often handle hundreds of millions of rows without a cluster. The comparison table in the next section spells out when those still win.
</aside>
Start with the smallest useful action: read a table and ask how many rows it has. That proves the catalog name, the cluster, and the spark session all work before you add joins.
flowchart LR
read["spark.read.table<br/>hyf.nyc_yellow.raw_trips"] --> df["DataFrame<br/>trips"]
df --> count["count()<br/>action"]
df --> show["filter + show(5)<br/>action"]
count --> n["one integer<br/>~128M"]
show --> sample["five sample rows"]
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class read,df,count,show,n,sample box;
The first line uses spark. That is Databricks' pre-created SparkSession (the Python entry point to Spark). You do not import or build it in a notebook; a laptop PySpark script would use SparkSession.builder.getOrCreate() instead. Helpers such as from pyspark.sql import functions as F you still import when you need them. The result of spark.read.table(...) is a DataFrame (PySpark) (a distributed table-like dataset on the cluster).
Type this in a notebook cell:
trips = spark.read.table("hyf.nyc_yellow.raw_trips")
zones = spark.read.table("hyf.nyc_yellow.raw_zones")
trips.printSchema()
print(trips.count()) # action: runs a job; expect roughly 128_000_000
After you run it, the notebook cell looks like this:
Databricks notebook cell: printSchema for raw_trips and count() returning 128202548
printSchema (print column names and types without scanning all rows) shows the schema without pulling row data to your laptop. count (return one integer row total) is an action (an operation that finally runs the Spark plan): executors scan the table; only the integer comes back to the driver.
Next, practice one transformation and one safe action. Transformations are lazy: they build a plan and do not run until an action needs a result (full explanation in the next section).
from pyspark.sql import functions as F
# Transformation: builds a plan, does not run yet
# F.col("payment_type") is a Column in that plan (not a Python bool).
# TLC: payment_type == 2 means cash.
cash_trips = trips.filter(F.col("payment_type") == 2)
# Action: runs the plan and prints a few sample rows
cash_trips.select("pickup_datetime", "total_amount", "payment_type").show(5)
After you run it, the notebook cell looks like this:
Databricks notebook cell: filter payment_type == 2 and show(5) sample rows
filter (keep rows that match a condition) and select (keep or rename columns) describe work. show (print a small sample of rows) is the action that finally runs them.
F is the usual short name for pyspark.sql.functions. F.col("payment_type") == 2 looks like ordinary Python, but it builds a Column expression for Spark's plan: compare the payment_type column to 2 later on the cluster. It is not a True/False value in your notebook process the way pandas df["payment_type"] == 2 is. On this TLC dataset, 2 means cash.
<aside>
⌨️ Hands on: Run both cells above. Use the screenshots as a guide for what success looks like: count() on the order of 128 million, and show(5) printing five rows without crashing.
</aside>