Week 13 - Big Data on Databricks

Welcome to Week 13! Every pipeline you have built so far runs on data that fits on one machine: the shared Azure Postgres database, a few million taxi rows, a single dbt project.

This week you move to the platform data teams reach for when the data outgrows one machine: Databricks. Databricks is built on the lakehouse pattern: one system for both cheap storage and powerful compute. Delta Lake is the table-layer foundation on top of those files.

Treat Databricks as a multifaceted platform, not a single tool. In one workspace you get catalogs and tables, notebook compute, SQL warehouses for dbt, and Jobs to schedule work without a laptop open. Missing the scheduling surface is missing part of what the platform is.

The through-line is the skill you already have. You are not learning a new job this week. You are running the analytics-engineering workflow from Week 10 at a scale where it starts to matter: the same dbt models, the same ref() calls, the same tests. The warehouse holds 100M+ rows instead of 56K. Then you schedule that build inside Databricks itself.

By the end of this week, you will run PySpark in a Databricks notebook, rebuild the Week 10 fct_trips mart as an incremental model against 100M+ rows of Delta data, and schedule that dbt build as a Databricks Job.

Further topics live on the optional Going Further page: lakehouse ACID deep dive, creating your own compute, Git-backed and multi-task Jobs, Databricks Genie (AI SQL), managed vs external tables, data governance with Unity Catalog, and an introduction to streaming.

How this week helps you get hired

NL data and analytics-engineering roles expect you to know when a pipeline has outgrown one machine, and what stays the same when you move it to a lakehouse platform:

explain why you would stay in dbt SQL vs reach for PySpark for a given task
port an existing dbt project to a new warehouse with only connection changes
configure incremental models and defend the boundary filter (> not >=)
schedule a Databricks Job with a dbt task on the SQL warehouse (and pause it on a shared bill)
treat cluster cost and secrets hygiene as production habits, not afterthoughts
recognise Databricks as more than notebooks: catalogs, SQL/dbt, and scheduled Jobs in the same workspace

Week 13 trains these skills on real scale (128M rows). The assignment PR is portfolio evidence that your Week 10 workflow survives a platform change.

For the NL-specific picture (posting mix, CV lines, interview phrasing), see Career relevance: Week 13.

Learning goals

Explain the lakehouse idea: why unified storage and compute beats a separate data lake plus warehouse, and what Delta Lake adds on top of files
Navigate the Databricks workspace, find tables via Unity Catalog (catalog.schema.table), and confirm data with SQL on the shared warehouse
Start a cluster, run PySpark in a Databricks notebook: the DataFrame API, lazy evaluation, and reading and writing Delta tables
Know when to reach for PySpark and when dbt SQL is the better tool
Connect dbt to the serverless warehouse, port the Week 10 fct_trips mart, and run it as an incremental model against 100M+ rows
Feel why incremental materialization matters: a first full build measured in minutes, later runs measured in seconds
Schedule the same dbt project as a Databricks Job (dbt task on hyf-dbt-warehouse) and explain how that relates to Week 12 Airflow

By the end of this week, you should be able to explain

when distributed compute earns its cost vs when a single Postgres server is still the right tool
why PySpark notebooks attach to a cluster but dbt runs on the serverless SQL warehouse
what is_incremental() and {{ this }} do on the second dbt build of fct_trips
when a Databricks Job is enough vs when you still want Airflow

Chapters

Supplementary

Slides (PDF)

Career relevance: Week 13

Glossary: Week 13

History of the Lakehouse

Going Further

Overview

Compute & notebooks

Lakehouse deep dive

Ingestion, DLT & medallion

Jobs & Asset Bundles

AI, Unity Catalog & quality

Streaming, internals & platforms