Assignment

The Scenario

Your team already runs the Week 10 taxi mart on Postgres. Leadership wants proof it scales on the shared Databricks lakehouse without rewriting the transformation logic. Your job is to explore the raw data in PySpark, port the dbt project, schedule the build as a Git-backed Databricks Job, and show in numbers why incremental materialization matters at 128M rows.

Read Gotchas & Pitfalls before you start. Most failed submissions trace back to one of those patterns (collect(), >=, or a committed token).

If you worked through Practice, Tasks 1, 2, and 3 reuse the same work with a formal submission layout.

Task 1: PySpark exploration (required)

In a Databricks notebook, read hyf.nyc_yellow.raw_trips into a PySpark DataFrame and answer two questions using only transformations and a single action (show(), not collect() on the raw table):

Which pickup borough has the most trips? (Join hyf.nyc_yellow.raw_zones on pickup_location_id = location_id to turn the location id into a borough, as PySpark in Databricks showed.)
What is the average total_amount per payment_type?

Then write two or three sentences on when you would reach for PySpark versus dbt SQL for a transformation.

<aside> 💡 Using AI to help: If your notebook errors on the join or aggregation, paste the traceback and your transformation chain into an LLM and ask it to spot the issue (⚠️ Ensure no tokens, connection strings, or PII are included!). Do not paste raw trip rows.

</aside>

Task 2: dbt on Databricks & incremental models (required)

Port your Week 10 dbt project to Databricks and run fct_trips as an incremental model.

Install dbt-databricks and configure a Databricks target in profiles.yml (token stored as an environment variable, never committed). Copy profiles.yml.example from the assignment repo and keep your real profiles.yml git-ignored.
Confirm dbt debug passes against Databricks.
Change fct_trips to materialized='incremental' with incremental_strategy='merge' and a correct unique_key.
Run dbt build --select fct_trips --full-refresh for the initial full build baseline, followed by dbt build --select fct_trips for the incremental rerun. Capture both wall-clock times and record them in task-2/WRITEUP.md with two or three sentences explaining why the second run was faster (name is_incremental() and the {{ this }} filter explicitly).
Run DESCRIBE HISTORY hyf.dev_yourname.fct_trips in Catalog Explorer or a notebook. Paste the resulting history table (showing CREATE OR REPLACE TABLE and MERGE operations) or include a screenshot link in task-2/WRITEUP.md.

Extra guidelines

The model logic should be unchanged from Week 10 apart from the config() block. If you find yourself rewriting the SQL, step back: the point is that transformation logic ports across warehouses.
Get the is_incremental() boundary right (>, not >=). A run that produces duplicates does not pass.

Task 3: Schedule a Git-backed dbt Job (required)

Production pipelines run unattended from version-controlled code. Build on Scheduling dbt Jobs by scheduling your ported dbt project directly from your GitHub repository fork:

Push your completed task-2/ dbt project to your GitHub fork of data-assignment-week-13 on branch main.
In Databricks Workflows → Jobs, create a Job named dev_yourname_fct_trips.
Add a dbt task type and select Git provider as the source:

Git repository URL: https://github.com/<your_username>/data-assignment-week-13.git
Git provider: GitHub
Git reference: main
Path / Project directory: task-2

Configure the task execution:

dbt commands: dbt deps followed by dbt build --select fct_trips
SQL warehouse: hyf-dbt-warehouse
Catalog / Schema: hyf / dev_yourname (your personal schema)

Click Run now and verify that the run completes with a green checkmark.
Add a schedule (for example, daily at 06:00 UTC) and immediately pause the trigger.
Fill in task-3/SCHEDULING.md with:

Your Databricks Job Run URL (copy from the run history address bar in Databricks).
Screenshots of your Job task configuration, the successful run log (showing dbt task output), and the paused schedule trigger.
Two or three sentences explaining when you would use Databricks Jobs versus Airflow for pipeline orchestration.

Task 4: Optional bonuses

These do not affect whether the assignment passes. Pick any you want for portfolio depth. Each bonus points at a Going Further section so you are not guessing the UI.

Easy - Workflows alerting (~5 min): On the Job from Task 3, configure an email notification for failure or completion. Put a screenshot in task-4/screenshots/job_notification.png and a one-line note in task-4/BONUSES.md. Reference: Job notifications.
Medium - Governance (~15-20 min): On your fct_trips table, run SHOW GRANTS, capture the lineage graph in Catalog Explorer, and write (do not need to run) the ALTER TABLE ... SET TAGS statement you would use for a hypothetical PII column. Paste evidence into task-4/BONUSES.md (and screenshots under task-4/screenshots/ if useful). An empty grants list on a personal schema is still a valid observation if you say so. Reference: Lineage & PII tagging.
Harder - Streaming (~20-30 min + cluster): Run the rate-source Structured Streaming demo, write results to a Delta table in your schema, describe what you observed in task-4/BONUSES.md, and stop the streaming query (then terminate idle cluster compute). Leaving a stream running burns the shared bill. Reference: Streaming: data in motion.
Harder+ - Local PySpark module + pytest (~45-90 min, laptop only): Copy the scaffold from pyspark_app_scaffold into task-4/pyspark_app/, implement the two Task 1 aggregations as pure functions in transform.py, and get uv run pytest -q green on the tiny fixture DataFrames. Paste the pytest output into task-4/BONUSES.md. Do not read Unity Catalog tables or start a cluster. Reference: Unit-testing a PySpark module locally.

Deliverables

Required:
task-1/: Your PySpark exploration notebook.
task-2/: Your dbt project (with profiles.yml.example, secrets excluded) and WRITEUP.md with both build times, incremental explanation, and Delta history (DESCRIBE HISTORY) proof.
task-3/: SCHEDULING.md with Job Run URL, Job screenshots (Git-backed config, green run, paused schedule), and your Jobs vs. Airflow comparison.
AI_ASSIST.md: Documenting at least one point where you used an LLM during this assignment (debugging, dbt Jinja, or Job scheduling).
Optional: Any Task 4 bonus evidence under task-4/ (BONUSES.md plus screenshots; for Harder+ also pyspark_app/ with green pytest).

<aside> ⚠️ Never commit your Databricks token or connection string. Keep them in environment variables, exactly as you did with the Postgres password in Week 12. A committed secret is an automatic fail and a real-world security incident.

</aside>

How to submit

Your work is submitted as a pull request back to the assignment repository:

Fork HackYourFuture/data-assignment-week-13. Your teacher may point you at a cohort fork in the HackYourAssignment organization instead; use whichever link they share.
Clone your fork locally, add your PySpark notebook to task-1/, your dbt project to task-2/, and your scheduling evidence to task-3/. Fill in task-2/WRITEUP.md, task-3/SCHEDULING.md, and AI_ASSIST.md.
Push your changes to branch main on your fork.
Schedule your Databricks Job pointing to https://github.com/<your_username>/data-assignment-week-13.git (branch main, path task-2), confirm a green run, and pause the trigger.
Open a pull request from your fork back to HackYourFuture/data-assignment-week-13.

Expected layout:

data-assignment-week-13/
├── task-1/
│   └── pyspark_exploration.ipynb    # or exported .py from Databricks
├── task-2/
│   ├── dbt_project.yml
│   ├── models/                      # your ported Week 10 models
│   ├── profiles.yml.example         # env_var placeholders only
│   └── WRITEUP.md                   # timings, explanation + Delta DESCRIBE HISTORY
├── task-3/                          # Git-backed Job scheduling
│   ├── SCHEDULING.md                # Jobs vs Airflow write-up + Job Run URL
│   └── screenshots/                 # Job config, green run, paused schedule
├── task-4/                          # optional bonuses only
│   ├── BONUSES.md                   # which bonuses + short notes
│   ├── screenshots/                 # e.g. job_notification.png, lineage
│   └── pyspark_app/                 # Harder+ only: local transforms + pytest
├── .env.example                     # DATABRICKS_* and DBT_SCHEMA template
├── AI_ASSIST.md                     # LLM usage notes
└── README.md

Package your pull request for review

Fill in every section of the PR template (it loads automatically when you open the PR on GitHub). A well-packaged PR includes:

Databricks Job Run URL at the top of the PR description.
Links or embedded screenshots of your PySpark output, WRITEUP.md timings, and Job execution log.
Clear confirmation that no secrets (profiles.yml, .env, tokens) were committed.

What good looks like

Your PySpark answers use a small aggregated result, not a collect() of the raw table.
The dbt project runs on Databricks with only the profile changed from Week 10.
The second dbt build is visibly faster than the first, WRITEUP.md explains why using is_incremental() and {{ this }} in your own words, and DESCRIBE HISTORY proves Delta Lake recorded the MERGE.
Your Databricks Job sources code directly from your public GitHub fork, runs successfully on hyf-dbt-warehouse, and has its schedule safely paused.
AI_ASSIST.md documents at least one critically evaluated LLM interaction.
Optional Task 4 bonuses (if any) live under task-4/ with a short note in BONUSES.md, any streaming demo includes proof you stopped the query, and Harder+ includes green local pytest output (fixture DataFrames only).

Stuck on collect(), cluster startup, or dbt debug tracebacks? Gotchas & Pitfalls is the first place to look.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

CC BY-NC-SA 4.0 Icons

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.