Week 13 - Big Data on Databricks
This page is optional. Nothing here is required for Week 13's learning goals or the assignment. Back to the Going Further overview.
In PySpark in Databricks you attached a shared class cluster from the notebook compute dropdown. That is the right path for the week: one warm cluster, predictable cost, no SKU guessing. Opening Compute in the sidebar is how you inspect cluster state, start or terminate machines, and eventually click Create compute yourself. This section is what that page and form are asking.
Create new compute form: policy, runtime, worker type dropdown, min/max workers, and DBU/h estimate
Do not create a personal cluster for class work unless your teacher says so. On the shared nonprofit subscription, extra clusters burn money and often hit SkuNotAvailable. Prefer the shared cluster (and hyf-dbt-warehouse for SQL/dbt).
When you are allowed to create one, the form knobs that matter:
| Field | What it means | Practical default |
|---|---|---|
| Policy | Admin-locked template (allowed node types, max workers, auto-term). | Use the class policy if one is offered (for example hyf-student-compute). Avoid Unrestricted on a shared bill. |
| Runtime | Databricks Runtime = Spark + libraries version. | An LTS runtime (long-term support), not the newest preview. |
| Photon | Faster engine for many SQL/DataFrame jobs; costs more DBU/h. | Fine for demos; turn it off if you are only learning and watching cost. |
| Worker type | Azure VM SKU (memory and cores per worker). | Smallest type that runs your job. Bigger is not automatically better. |
| Min / Max workers | Autoscaling range. Each worker adds cost while the cluster is up. | Start with Single node (driver only) for learning; add workers only when a job is slow and parallel. |
| Auto-termination | Shuts the cluster down after idle minutes. | Keep it short (for example 10–20 minutes). Idle clusters still bill. |
The summary panel on the right (memory, cores, DBU/h) is the cost forecast. Read it before you click Create.
Cluster vs warehouse still holds. Creating compute here builds an all-purpose cluster for notebooks and PySpark. SQL and dbt still use a SQL warehouse (Create under SQL → SQL Warehouses), not this form. Same rule as the required chapters: notebooks → cluster; SQL/dbt → warehouse.
In PySpark in Databricks the required path attaches your notebook to the shared classic cluster. Databricks also offers serverless compute for notebooks: pick Serverless in the notebook compute dropdown and run Spark/PySpark without starting a classic cluster yourself (Microsoft Learn).
Three names that look alike:
| Name in the UI | What it runs | Week 13 use |
|---|---|---|
| Serverless (notebook dropdown) | Managed Spark for notebook cells | Optional fallback / exploration |
| Shared classic cluster | Spark with visible driver + executors | Required Ch3 path |
SQL warehouse (hyf-dbt-warehouse) |
SQL and dbt | Ch2 count + Ch4 dbt |
Serverless notebooks are still Spark. They are not the SQL warehouse. Do not call attaching a notebook to hyf-dbt-warehouse "serverless notebooks" (that only runs SQL cells, not Python).
Why the chapter keeps the classic cluster. You attach a real cluster with a visible driver plus executors, see why idle minutes cost money, and the Spark story matches the machines underneath. Serverless hides that operational surface. Starting, terminating, and creating clusters on the Compute page is optional depth in Creating your own compute.
When it is useful: if the shared cluster is stuck on a long start or fails with SkuNotAvailable, and your teacher says the workspace has serverless interactive compute enabled, attach Serverless and keep going. New notebooks may default to Serverless when nothing else is selected; for class work, prefer the shared cluster when it is Running.
Limits to know: serverless notebook compute does not expose the classic Spark UI the same way a cluster does. Long-running cells can hit an execution timeout (workspace default is often around 2.5 hours). Cost still matters: you pay for what you run, even without a cluster you started by hand.
In PySpark in Databricks you use a notebook the way many data scientists do on Databricks: explore interactively, keep state between cells, share a page. That is a strength for discovery. It is a weak default for production.
A large part of a data engineer's workload is taking a notebook that "works when I click Run All" and turning it into something a platform can run without a human in the loop: a .py module, a Databricks Job, a scheduled dbt model, or a small package with tests. Week 4 said the same thing for local Jupyter: notebooks for research, scripts for production. On a lakehouse team the hand-off is often DS notebook → DE job.
Why migrate. Notebooks are hard to unit-test, easy to break with out-of-order cell runs, awkward in code review when outputs are huge, and expensive if someone leaves a cluster attached while "just checking one cell." Applications and jobs give you parameters, retries, logging, CI, and a clear entry point (main, a Job task, an Airflow/ADF trigger).
A practical migration shape (not required this week; a map for later jobs):
df still exists").collect() on large frames for show(), writes, or aggregations that stay on the executors. Prefer writing a Delta table over pasting a giant grid into the notebook.dev / prod) become Job parameters or CLI args, not hard-coded cells.What stays a notebook. Ad hoc investigation, one-off QA, teaching demos (this chapter), and early DS prototypes. Do not feel guilty for using a notebook to find the answer. Feel responsible for not leaving the only copy of a critical pipeline as twelve untitled cells.
The next section is the hands-on version of step 5 above: extract Task 1 logic into a local .py module and prove it with pytest on tiny fixtures (no Unity Catalog, no class cluster).
Optional stretch for students who finished Tasks 1–3 and want a DE-shaped portfolio piece. This is the concrete version of "extract functions + add a test on a small fixture" from Migrating notebooks to applications.
Goal. Turn the Task 1 borough / payment aggregations into pure Python functions in a .py file, then prove them with pytest on tiny fixture DataFrames. You never open Databricks for this bonus. You never call spark.read.table(...).
Why fixtures, not the 128M table. Unit tests should be fast, free, and deterministic. A five-row DataFrame is enough to prove a join and a groupBy. Hitting Unity Catalog from your laptop needs Databricks Connect or a Job: that is a different project, and it is out of scope here.
task-4/pyspark_app/
├── transform.py # pure functions: DataFrame in → DataFrame out
├── tests/
│ └── test_transform.py # SparkSession fixture + tiny rows
├── pyproject.toml # pyspark + pytest
└── .gitignore
Starter files live in this repo under pyspark_app_scaffold. Copy that folder into your assignment fork as task-4/pyspark_app/.
Step 1: Copy the scaffold
From a clone of this curriculum repo (or a zip your teacher shares):
# from your data-assignment-week-13 fork
mkdir -p task-4
cp -R "<path-to-datatrack>/Data Track/Week 13/assets/pyspark_app_scaffold" task-4/pyspark_app
cd task-4/pyspark_app
On GitHub you can also browse pyspark_app_scaffold and download the files by hand into task-4/pyspark_app/.
Step 2: Install and confirm the tests fail
uv sync
uv run pytest -q
Expect NotImplementedError until you fill in transform.py. That is the fail-by-default pattern from earlier weeks.
Step 3: Implement the two functions
In transform.py, replace the raise NotImplementedError(...) bodies. Keep the functions pure: they take DataFrames and return a DataFrame. Create SparkSession only in the test fixture (already done) or in a thin __main__ runner if you add one later.
Hints that match Task 1 / Ch3:
top_pickup_borough: join on pickup_location_id == location_id, groupBy("borough"), count, orderBy descending, limit(1). Alias the count as trip_count.avg_total_by_payment_type: groupBy("payment_type"), avg("total_amount") aliased avg_total_amount.Import helpers as needed: from pyspark.sql import functions as F.
Step 4: Re-run tests
uv run pytest -q
Success: both tests green. Paste the command output into task-4/BONUSES.md (two or three lines is enough) and note that you used local fixtures, not Unity Catalog.
PySpark on your laptop needs a JDK (Java 11 or 17) and Python 3.11 to 3.13. The scaffold pins pyspark to 3.5.x (same major line as the class Databricks Runtime). You do not need Databricks Connect, a cluster, or Key Vault for this bonus.
macOS (Homebrew):
brew install openjdk@17
echo 'export PATH="$(brew --prefix openjdk@17)/bin:$PATH"' >> ~/.zshrc
echo 'export JAVA_HOME="$(brew --prefix openjdk@17)"' >> ~/.zshrc
source ~/.zshrc
java -version
Check before pytest: if java -version fails, Spark will fail too (JAVA_GATEWAY_EXITED / "Unable to locate a Java Runtime"). Fix Java first, then uv run pytest -q again.
hyf.nyc_yellow.raw_trips from the moduletask-4/<aside>
📝 Optional assignment bonus (Harder+): Assignment Task 4 asks for task-4/pyspark_app/ with green pytest output noted in BONUSES.md. Use the scaffold under pyspark_app_scaffold.
</aside>
spark fixture pattern used in the scaffold.