Week 13 - Big Data on Databricks

The lakehouse idea

Workspace & Unity Catalog

PySpark in Databricks

dbt on Databricks

Scheduling dbt Jobs

Practice

Assignment

Gotchas & Pitfalls

Slides (PDF)

Career relevance: Week 13

Glossary: Week 13

History of the Lakehouse

Slides (PDF)

The slide deck for the Week 13 mid-week in-class session. Students are already on the shared Databricks workspace. Structure: progress check → Exercises A–E frame → lock-in / live Exercises A–E → assignment close. Live exercises use workspace artifacts (SQL editor, notebook, local dbt project, Jobs UI), not a reference-repo git switch walk. The reference branch week-13-ch-4-dbt-solution is catch-up / teacher Job demo only.

Present in the browser (HTML)

Interactive present-mode deck (arrow keys / click). Includes the Ch4 compile-fork and MERGE mini-sim widgets that the PDF cannot show:

https://hackyourfuture.github.io/Learning-Resources/data-track/embeds/week-13-slides/week_13__presentation.html

Download (PDF)

week_13__presentation.pdf

week_13__presentation.pdf

What's in it

Teacher script, pacing, and quiz cribs: GitHub-only week_13__lesson_plan.md.

To regenerate after editing the Marp source: just slides-rebuild 13 from the repo root (PDF + HTML + slide-hash comments). Teachers present iframe slides with just slides-present 13. While iterating on the deck locally, just slides-dev 13 watches the Marp file and reloads the browser after each HTML rebuild (~1 min per save).

Exercises (live in class)

Ex / live Section Start Success check Feeds
Exercise A §2 Ch2 SQL editor + hyf-dbt-warehouse count(*) ≈ 128,202,548 Ch2 lock-in
Exercise B §3 Ch3 Python notebook + shared cluster Top pickup borough + avg total_amount by payment_type via show() Task 1
Exercise C §4 Ch4 Own dbt project + warehouse connection details dbt debugAll checks passed! Task 2
Exercise D §4 Ch4 Incremental fct_trips Two timings; second clearly faster Task 2
Exercise E §5 Ch5 Workflows → Job (demo then own fork) Green Run now + paused schedule + Run URL Task 3

Live Exercises A–E cover the workspace spine. Ch1 is teacher lock-in only (Hands-on stays in the chapter). Catch-up only (not the class spine): nyc-taxi-dbt-reference branch week-13-ch-4-dbt-solution if the port is broken. Async Practice exercises remain on the Practice page.


The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

CC BY-NC-SA 4.0 Icons

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.