Week 13 - Big Data on Databricks
This page is end-of-week synthesis, not a fresh set of drills. Work through it after you have completed the five content chapters, because each exercise reuses the artifacts you built as you went: the Databricks workspace access from Workspace & Unity Catalog, the notebook skills from PySpark in Databricks, the ported dbt project from dbt on Databricks, and the Job wiring from Scheduling dbt Jobs. The exercises build on one shared thread (the taxi data on Databricks) rather than starting over each time.
<aside>
📦 Reference repo: Clone nyc-taxi-dbt-reference once. Branch week-13-ch-4-dbt is the starter branch for Chapter 4, and week-13-ch-4-dbt-solution is the finished Databricks port. Each practice exercise below has a matching week-13-practice-exercise-* branch with an EXERCISE.md at the repo root and a -solution branch to compare against when you are stuck.
</aside>
git clone <https://github.com/lassebenni/nyc-taxi-dbt-reference.git>
cd nyc-taxi-dbt-reference
git switch week-13-practice-exercise-pyspark-exploration # then read EXERCISE.md
<aside> 📝 Practice exercises are recommended but optional. They are the fastest way to turn this week's reading into skill you can show in an interview.
</aside>
Before you start, make sure your dbt on Databricks project runs against Databricks (dbt debug passes) and your fct_trips model is built. Several exercises assume it exists. If your port is broken, git switch week-13-ch-4-dbt-solution in the reference repo gives you a known-good starting point.
<aside> 💭 If you get blocked for more than 20 minutes, write what you tried, switch to the next exercise, and ask for help. This is professional behavior, not failure.
</aside>
| Exercise | Primary Chapter | What you produce |
|---|---|---|
| 1: Unity Catalog & Delta history | Workspace & Unity Catalog | SQL query results inspecting table commits and access grants |
| 2: PySpark exploration | PySpark in Databricks | A Databricks notebook with two aggregated show() results |
| 3: Incremental in action | dbt on Databricks | Two dbt build timings plus a short WRITEUP.md |
| 4: Create and schedule a dbt Job | Scheduling dbt Jobs | A green manual run of dev_yourname_fct_trips and a paused trigger |
| 5: Honest write-up | The lakehouse idea | A half-page lakehouse synthesis in WRITEUP.md |
Attempt each exercise yourself first, then check against the reference branches (week-13-practice-exercise-*-solution) when applicable. Do not open solution branches before trying: they are for catching up, not for skipping the thinking.
Concepts: 3-level namespace (hyf.nyc_yellow.raw_trips), DESCRIBE HISTORY, Unity Catalog privileges (SHOW GRANTS).
Primary chapter(s): Workspace & Unity Catalog
In the Databricks SQL Editor (or in a Databricks notebook cell running %sql), run two diagnostic queries against your class table:
Step 1: Inspect the Delta transaction log history for hyf.nyc_yellow.raw_trips:
DESCRIBE HISTORY hyf.nyc_yellow.raw_trips;
Look at the version, timestamp, and operation columns. What operation created or appended this table?
Step 2: Inspect the Unity Catalog access controls on the table:
SHOW GRANTS ON TABLE hyf.nyc_yellow.raw_trips;
Success criteria: You can identify the Delta table version history and state in one sentence who owns or has SELECT grants on hyf.nyc_yellow.raw_trips.
Concepts: PySpark DataFrames, join, groupBy, show() versus collect(), PySpark versus dbt SQL.
Primary chapter(s): PySpark in Databricks
<aside>
📦 Branch: week-13-practice-exercise-pyspark-exploration: starter notebook at notebooks/pyspark_exploration.py with TODO markers; compare against week-13-practice-exercise-pyspark-exploration-solution.
</aside>
In a Databricks notebook, read hyf.nyc_yellow.raw_trips into a DataFrame and answer, using only PySpark transformations and a single action:
hyf.nyc_yellow.raw_zones on pickup_location_id = location_id to turn the location id into a borough, as PySpark in Databricks showed.)total_amount per payment_type?Success criteria: Two small aggregated tables displayed with show() (not collect() on the raw frame), plus a sensible PySpark-versus-dbt answer that names at least one concrete trigger from PySpark in Databricks.
Concepts: incremental materialization, is_incremental(), {{ this }}, build timing.
Primary chapter(s): dbt on Databricks
<aside>
📦 Branch: week-13-practice-exercise-incremental-timing: fill in WRITEUP.md after timing two dbt build runs; sample answers on week-13-practice-exercise-incremental-timing-solution.
</aside>
Run dbt build --select fct_trips twice and record the wall-clock time of each run. Write two or three sentences explaining, in your own words, why the second run was faster. Reference is_incremental() and {{ this }} explicitly.
This is the exercise to be able to talk through in an interview: it is the concrete story behind "I understand incremental models."
Success criteria: Both run times recorded in WRITEUP.md, the second run visibly faster than the first, and your explanation names is_incremental() and the {{ this }} filter (not "the cluster stayed warm").
Concepts: Databricks Workflows, dbt task on hyf-dbt-warehouse, manual execution, paused triggers.
Primary chapter(s): Scheduling dbt Jobs
Using the Git provider source pattern from Scheduling dbt Jobs:
Step 1: Open Databricks Workflows → Jobs and create a new Job named dev_yourname_fct_trips.
Step 2: Configure a dbt task type with Source set to Git provider (https://github.com/lassebenni/nyc-taxi-dbt-reference.git, branch week-13-ch-4-dbt-solution), set SQL Warehouse to hyf-dbt-warehouse, and set Catalog / Schema to hyf and your schema.
Step 3: Set the dbt commands to dbt deps followed by dbt build --select fct_trips.
Step 4: Click Run now and verify that the run completes with a green checkmark.
Step 5: Add a schedule (for example, daily at 06:00 UTC) and then pause the trigger immediately.
Success criteria: One successful manual run recorded under the Job's Runs tab, and the schedule trigger status shows Paused.
Concepts: lakehouse mental model, dbt portability, Databricks Jobs versus Airflow, week synthesis.
Primary chapter(s): The lakehouse idea (synthesis across the week)
<aside>
📦 Branch: week-13-practice-exercise-lakehouse-writeup: WRITEUP.md scaffold with section prompts; sample tone on week-13-practice-exercise-lakehouse-writeup-solution.
</aside>
Write a short note (half a page) you could send to a teammate explaining what you did this week:
This is portfolio material. Clear writing about your own work is a rarer skill than the code itself.
Success criteria: A completed WRITEUP.md that answers all prompts in plain language, mentions Delta or the transaction log at least once, explains Jobs versus Airflow in one sentence, and could be read by a teammate who was not in the class.
If you finish the core exercises and want more, the Going Further page has two hands-on optional topics on the tables you just built: read table lineage with Unity Catalog, and run the rate-source Structured Streaming demo. Neither is required for the week.
Next up: Assignment. Exercises 2, 3, and 4 are deliberate warm-ups for the required tasks there; if you completed them, you are most of the way to a passing submission.
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.