Week 13 - Big Data on Databricks
Every tool this week has a way to bite you. These are the ones that catch students most often. Read them before the Assignment; each one has cost someone real time or real money.
Each entry links back to the chapter that teaches the correct pattern. Use this page as an index when something misbehaves.
collect() on a large DataFrame crashes the drivercollect() pulls the entire result onto the single driver machine. On a small aggregated result that is fine, but on a raw or lightly-filtered 100M-row table it tries to fit everything into one machine's memory and the job dies with an out-of-memory error.
Fix: aggregate or filter first, and use show() (which displays only the first rows) for inspection. Reserve collect() for results you know are small. See PySpark in Databricks.
A notebook can use the serverless SQL warehouse for SQL cells (same idea as the SQL editor). Python and PySpark need a cluster (driver + executors). If you leave the notebook on the warehouse and run Python, cells fail or complain about a missing Spark context.
Fix: for SQL exploration in the notebook, hyf-dbt-warehouse is fine. Before PySpark cells, attach hyf-class-cluster from the notebook compute dropdown and wait until Connected. Keep dbt on the SQL warehouse. See PySpark in Databricks.
A cell that runs instantly and produces no output usually means the notebook is not attached to compute, or the cluster is still starting. For SQL, attach the warehouse; for Python, attach a running cluster.
Fix: attach the notebook to the right compute and wait for it to report as ready before you run cells. See PySpark in Databricks.
>= in an incremental filter creates duplicatesAn incremental model that filters with pickup_datetime >= (select max(pickup_datetime) from {{ this }}) re-reads the boundary rows it already loaded, inserting duplicates on every run.
Fix: use > (strictly greater) when the timestamp advances, and make sure your unique_key is genuinely unique. A trip_id built by hashing a few columns is not perfectly unique: two truly identical trips hash to the same id and a merge will dedupe them. That is usually fine, but know it is happening. If you already have duplicates, a full refresh (dbt build --select fct_trips --full-refresh) rebuilds the table cleanly. See dbt on Databricks.
Point the Week 10 dbt project at the 128M-row Databricks table and tests that passed on the clean 57K sample start failing. The accepted_values test on payment_type fails because the yellow data carries a code (0) the green sample never had, and the "pickup before dropoff" test flags a few thousand genuinely dirty rows.
Fix: this is real analytics-engineering work, not a porting bug. Read what the test is telling you, then either widen the rule to match reality (add 0 to accepted_values) or lower its severity to warn when it is known dirty source data. A test failing on real data at scale is the test doing its job. See dbt on Databricks.
dbt debug crashes before it connectsIf dbt-databricks blows up with a Python traceback (something about mashumaro and an "unserializable field") before it even reaches the connection test, you are almost certainly on too new a Python. dbt does not yet support the newest Python releases.
Fix: run dbt on Python 3.11 or 3.12. With uv: uvx --python 3.11 --from dbt-databricks dbt debug. See dbt on Databricks.
A dapi... token in profiles.yml, .env, or a notebook cell ends up in Git history forever. Anyone with repo access (and any LLM you paste the file into) can use it. The assignment autograder and your teacher both treat a committed secret as an automatic fail.
Fix: keep the token in environment variables only (DATABRICKS_TOKEN), read via env_var() in profiles.yml.example, and git-ignore the real profiles.yml and .env. If you already pushed a token, tell your teacher so it can be rotated. See dbt on Databricks and Assignment.
A daily schedule across a cohort of Jobs will wake hyf-dbt-warehouse even when nobody is learning. That is real spend with no new skill.
Fix: after you attach a schedule in Scheduling dbt Jobs, pause the Job (or the schedule) unless your teacher asks you to leave it on. Manual Run now is enough to prove the wiring. The assignment Scheduling Extra expects paused-schedule evidence.
fct_trips like everyone elseThe Workflows list is shared. Unprefixed names collide and make support impossible.
Fix: name Jobs dev_yourname_fct_trips (same namespacing habit as Week 12 DAG ids).
A Databricks cluster bills for every minute it runs, whether or not you are using it. A cluster left running overnight is money spent for nothing, and a whole class doing it adds up fast.
Fix: the shared workspace auto-terminates idle clusters after about 20 minutes, but do not rely on it alone. Let the cluster stop when you finish, and never disable auto-termination. See The lakehouse idea.
On the class subscription, starting a cluster is slow: the first start can take ten minutes or more while Azure allocates the machine, and on a busy day it can fail outright with a capacity error (SkuNotAvailable). This is not your code: the shared nonprofit subscription is running short of the specific machine type.
Fix: start the cluster (or ask your teacher to pre-start the shared one) before you need it, and do not panic if the first attempt is slow. If it fails with a capacity or SKU error, retry, or use serverless compute where available. Do not sit re-running a cell against a cluster that has not finished starting. See Workspace & Unity Catalog.
Serverless SQL warehouse compute runs in the Databricks account, not directly in the resource groups you see in the Azure portal, so its cost and controls live in the Databricks admin surface.
Fix: check spend and cluster policy in the Databricks workspace admin settings, not only the Azure cost view. Your teacher manages the workspace-level budget alert. See Workspace & Unity Catalog.
Streaming adds permanent complexity: always-on compute, checkpoints, harder debugging. It is tempting to stream something that is really needed once a day.
Fix: default to batch. Choose streaming only when low latency is a genuine, hard requirement. If the answer to "how fresh does this need to be?" is measured in hours, it is a batch job. The optional streaming demo lives on Going Further.
Before you open the Assignment, walk through each gotcha against the work you already have. For each row, answer in one line: does it apply? (yes / no / n/a) If yes, does your current code or repo handle it? (yes / no)
collect() on a raw table : PySpark notebook uses show() on aggregated results only>= in incremental filter : fct_trips uses > at the boundarydbt debug Python version : running on 3.11 or 3.12dapi string, no real profiles.yml or .env in gitPaste the ten-line checklist into a draft comment on your PR if it helps. Not graded, but it is the difference between reading the gotchas page and using it to catch bugs before your teacher does.
is_incremental(), unique_key, and merge strategies behind the >= gotcha.env_var() pattern for tokens.In the Assignment, you submit the PySpark notebook and the ported dbt project as a pull request. Walk this gotchas list once more before you open the PR.
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.