Week 13 - Big Data on Databricks
Every pipeline you have built so far shares one quiet assumption: the data fits on one machine. The shared Azure Postgres database, the taxi dataset, your dbt project, all of it runs comfortably on a single server. That assumption holds for a surprising amount of real work, and when it holds, you should not reach for anything heavier.
This chapter is about what changes when that single-machine assumption stops holding. Data teams then reach for Databricks, built on an idea called the lakehouse (one copy of data in cheap storage, with warehouse-style tables on top). You will not write any code here. The goal is to build the mental model. When you open a Databricks notebook later this week, you should understand what you are looking at and why it is shaped the way it is.
By the end of this chapter, you should be able to:
A single machine has a fixed amount of memory and a fixed number of CPU cores. The shared Azure Postgres you have been using is one VM doing that job: one set of CPUs, one pool of RAM, one disk. Postgres can store data larger than memory and still answer queries, but every scan, join, and sort still runs on that one machine. A query that scans a billion rows runs as fast as those CPUs allow and no faster.
Distributed computing (spreading data and work across many machines) removes that ceiling. Ten machines scanning a billion rows each handle a hundred million, in parallel. The query finishes roughly ten times faster.
That power is not free. Coordinating ten machines adds overhead: the data has to be split up, partial results have to be shuffled between machines and combined, and the whole cluster costs money for every minute it runs. For a 56K-row table, that overhead makes a distributed engine slower than one Postgres VM, not faster.
<aside> 💡 The honest rule: reach for distributed computing when the data genuinely does not fit or the job is genuinely too slow on one machine, not because it sounds more impressive. Most data problems are not big-data problems.
</aside>
Keep that rule in mind as you size up your own past work.
<aside> ⌨️ Hands on: Think of one pipeline you have built in this track and estimate its data size (rows times columns, roughly). Would one Postgres VM like the shared Azure database handle that workload comfortably? For almost everything in this course, the answer is yes, and that is the point: you are about to learn the tool for the cases where the answer is no.
</aside>
Before the lakehouse, teams usually ran two separate systems, and the split caused real pain.
A data lake (cheap object storage for raw files at scale) is a folder tree of files, not a database. On Azure, that is ADLS Gen2 (Azure Data Lake Storage Gen2: scalable object storage). The AWS equivalent you may have heard of is S3. Picture your taxi pipeline dumping one Parquet file per day under raw/trips/2024-01-15.parquet, raw/trips/2024-01-16.parquet, and so on. That is cheap and scales to any size, but the lake does not know those files are one table. There are no transactions and no schema checks. If a load job is still writing today's file when a dashboard job opens the same path, the reader can see a half-written dataset. If two jobs write the same path at once, they can corrupt each other.
<aside> 📘 Reminder: Parquet is the columnar analytics file format from Reading file formats (Week 3) and Writing data (Week 4). This week treats a folder of those files as the lake; Delta adds the table layer on top.
</aside>
Querying a lake means pointing a script or tool at a path, not at a named table. A plain Python job might do pd.read_parquet("abfs://…/raw/trips/") and scan every matching file under that folder. There is no built-in query service inside the lake itself: storage holds the files, and your process (or another engine) reads them. That works for exploration, but each job has to rediscover which files exist, what columns they have, and whether a write finished.
A data warehouse (an analytics database with real tables and fast SQL) gives you transactions and named tables (Snowflake, Google BigQuery, Microsoft Fabric). You query it the way you query Postgres: SELECT * FROM trips WHERE pickup_date = '2024-01-15'. The warehouse owns column types and a built-in SQL engine; you never point it at a folder of files.
That is the contrast with the lake. On the lake you open a path and hope the files under it form one usable dataset. In the warehouse you open a table the engine already knows about. The catch is that you load a copy of the data into those tables: you land files in object storage like ADLS, then copy curated rows into the warehouse. Since most companies also keep raw data in a lake, that second copy costs money to store, and it can drift out of sync with the lake.
flowchart TB
subgraph lake["Data lake"]
path["Open a path<br/>abfs://…/raw/trips/"]
files["Parquet / JSON files<br/>cheap, any size"]
path --- files
miss["No tables · no transactions<br/>no schema checks"]
files --- miss
end
subgraph wh["Data warehouse"]
sql["Open a table<br/>SELECT * FROM trips"]
tables["Named tables + types<br/> + schema enforcement + fast SQL"]
sql --- tables
cost["Second copy of the data<br/>storage + sync drift"]
tables --- cost
end
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class path,files,miss,sql,tables,cost box;
So teams stored everything twice: raw in the lake, curated in the warehouse, with a pipeline copying between them.
The lakehouse collapses the two into one. You keep the cheap object storage of the lake. On top of the files you add a table layer that gives you the transactions, schema, and SQL of a warehouse. One copy of the data, warehouse guarantees, lake economics.
flowchart TB
subgraph old["The old split: two systems, two copies"]
raw1["Raw files<br/>(data lake)"] -->| copy job| wh2["Curated tables<br/>(warehouse)"]
end
subgraph new["The lakehouse: one system"]
files2["Files in object storage<br/>(ADLS Gen2)"]
table["Table layer:<br/>transactions + schema + SQL"]
files2 --- table
end
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class raw1,wh2,files2,table box;
When people say a warehouse (or a lakehouse) gives you "real tables," they usually mean ACID writes: Atomic, Consistent, Isolated, and Durable. You already relied on all four on the shared Azure Postgres.
| Holds? | Means | Taxi example on Postgres | |
|---|---|---|---|
| Atomic | Yes | All-or-nothing | Load of 10k Jan 15 trips crashes mid-insert: 0 new rows, not 4k |
| Consistent | Yes | A commit only succeeds if declared DB constraints still hold | Primary key trip_id cannot be null: an insert without a trip id is refused |
| Isolated | Yes | Concurrent work does not see half-done writes | Dashboard select during the load sees yesterday or full Jan 15, never half |
| Durable | Yes | A commit survives a crash | After commit, restart Postgres: those trips are still on disk |
Cloud warehouses keep strong A/I/D on table writes. C is often weaker than Postgres (foreign keys may be informational only). That is the "Mostly" in the eras table below. Teams still paid for a second copy of the data to get those table-write guarantees.
| Era | ACID? | Reality |
|---|---|---|
| Database (Postgres) | Yes | One engine on one machine; commit or roll back; constraints often enforced |
| Data warehouse | Mostly | Strong A/I/D on table writes; C often weaker than Postgres (FKs may be informational) |
| Data lake | No | Just files on object storage; half-written days and colliding writers are possible |
| Lakehouse | Yes | Delta transaction log restores warehouse-style A/I/D on the lake files |
The lakehouse pitch is not "forget ACID." It is "keep ACID without the warehouse copy."
<aside> 📚 For the letter-by-letter taxi examples on a cloud warehouse, a plain lake, and Delta (and how dbt tests differ from DB constraints), see the optional Lakehouse deep dive.
</aside>
A folder of Parquet files is storage, not a table. You cannot safely ask "give me all trips" when today's file is still being written. You cannot UPDATE one row without rewriting a whole file. You cannot tell a dashboard "use yesterday's version" after a bad load. That is why the lakehouse adds a table layer on top of the files: something that tracks which files belong to which table version, which schema those files must follow, and which writes have finished.
Without that layer you are stuck with the old split: keep cheap files in the lake, then copy curated data into a warehouse so analysts get real tables. The table layer's job is to give you those warehouse guarantees on the lake files themselves, so you do not need the second copy.
On Databricks, that table layer is Delta Lake (Parquet files plus warehouse guarantees). Delta is open-source software developed by Databricks: it extends Parquet with a file-based transaction log for ACID writes and scalable metadata. A Delta table is a folder of Parquet files plus a transaction log (an ordered record of every change). That log is what turns a pile of files into a real table.
flowchart TB
subgraph plain["Plain Parquet folder"]
pfiles["Parquet files on ADLS"]
pmiss["No ACID · no schema checks<br/>no time travel · no MERGE"]
pfiles --- pmiss
end
subgraph delta["Delta table"]
dfiles["Parquet files on ADLS"]
dlog["Transaction log<br/>(ordered record of changes)"]
dfiles --- dlog
dgive["ACID · schema enforcement<br/>time travel · atomic MERGE"]
dlog --- dgive
end
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class pfiles,pmiss,dfiles,dlog,dgive box;
The transaction log buys you four things plain files cannot give you:
hyf.nyc_yellow.raw_trips when a dashboard queries the same table. With Delta, the dashboard either sees the complete January 15 load or still sees yesterday's version. It never reads a half-written day of taxi trips, and a second writer cannot corrupt the table mid-load.fare_amount and tip_amount are decimals on raw_trips. A bad file that tries to write "N/A" into fare_amount, or an extra mystery column the table does not define, is rejected. The broken batch does not land silently next to good yellow-taxi rows.fct_trips with a bad filter and Manhattan tip totals look wrong. Because the log kept the previous version, you can query fct_trips as it looked before that load, compare tip sums, and restore or re-run without digging through backup Parquet folders.fct_trips already has January 14. Today's batch brings new January 15 trips and a corrected tip_amount for one January 14 trip. MERGE updates that one existing row and inserts the new ones in a single transaction. That is the primitive that makes the incremental dbt models in dbt on Databricks fast, and it is why Delta, not plain Parquet, is the foundation for this week.<aside> 📚 For who coined "lakehouse", how the name combines lake and warehouse, and how Delta relates to Iceberg and Hudi, see the optional History of the Lakehouse.
</aside>
You are not starting over. This week runs the workflow you already know on a bigger engine:
fct_trips mart you built in Week 10 and run it on Databricks. Same ref() calls, same tests, same YAML. The only thing that changes is the connection profile. That is the whole portability argument from Week 10, proven.<aside> 💡 Using AI to help: If you paste a Databricks error or concept snippet into an LLM to understand it faster, treat the workspace URL, connection string, and any access token as secrets. ⚠️ No real data, no PII, no tokens.
</aside>
Databricks compute costs money for every minute a cluster runs. The shared HYF workspace is configured with auto-termination (an idle cluster shuts itself off) and a spending alert (a warning when spend crosses a set threshold). The habit still matters: when you finish a hands-on, let the cluster terminate. You will see this reminder again in Gotchas & Pitfalls, because forgetting it is the single most common way a class runs up a bill.
https://lasse.be/simple-hyf-teach-widget/mcq.html?bank=week_13_ch1_intro_lakehouse_quiz&embed=1
If the lake versus warehouse versus lakehouse split still feels abstract, this short video walks through it from a different angle.
https://www.youtube.com/watch?v=myLiFw9AUKY
The next chapter gets you into the Databricks workspace and its compute. There is no state to set up yet, so this is a mental-model check. You are ready when:
<aside> 📝 Practice: Later this week, Practice Exercise 3 asks for a short lakehouse write-up that reuses this chapter's mental model. It is optional synthesis after you finish the content chapters, not homework for right now.
</aside>