Week 13 - Big Data on Databricks
This page answers two questions students ask every week: why am I learning this, and how does it help me find a job?
It is scoped to Week 13 content (the lakehouse, Databricks, PySpark, and dbt at scale). Other weeks' career pages each cover their week's tool. Generic NL junior data-career content (salary bands, day-to-day work, what employers do not expect from juniors) lives in one shared page across the curriculum and is not repeated here.
The numbers below are measured from a private NL jobs crawl (active listings as of July 2026): skill tags plus title/description mentions, broken down by role. Analytics-engineer cells are noisier (small sample, ~30 active titles). Round figures; treat them as market signal, not precision.
| Tool / skill | Data engineer | Analytics engineer | Data scientist | Data analyst |
|---|---|---|---|---|
| Databricks | Common (~36%) | Common (~27%) | Occasional (~19%) | Occasional (~5%) |
| Apache Spark / PySpark | Common (~27%) | Occasional (~10%) | Occasional (~20%) | Rare (~2%) |
| dbt | Occasional (~19%) | Very common (~40%) | Rare (~1%) | Rare (~2%) |
| Delta Lake / lakehouse | Occasional (~22%) | Occasional (~20%) | Rare (~3%) | Rare (~5%) |
What this means in practice: Databricks is a credible answer when a role mentions "Spark", "lakehouse", or "Delta", but it is not the only platform NL teams use. The portable skill this week teaches is analytics engineering at scale: same dbt project, bigger warehouse, incremental models that earn their keep past tens of millions of rows, plus a Git-backed Job so the build can run unattended.
Interview vocabulary tip: many lakehouse teams call raw / staging / marts bronze / silver / gold (medallion architecture). That is naming, not a new modelling skill. Optional depth: Medallion architecture.
Week 13 teaches both on purpose. The crawl shows they are complementary skills that rarely travel together, not interchangeable keywords.
Corpus-wide (~3,800 active listings): Spark/PySpark ~5% (188 jobs), dbt ~3% (132 jobs). Only 40 jobs name both; 148 name Spark without dbt; 92 name dbt without Spark. Explicit "PySpark" alone is rarer (~2% / 91 jobs); many postings say "Spark" and mean the Python API.
| Role | Spark / PySpark | dbt | Both | Who wins? |
|---|---|---|---|---|
| Data engineer | ~27% (109) | ~19% (76) | 35 jobs | Spark ahead; meaningful overlap |
| Analytics engineer | ~10% (3) | ~40% (12) | 2 jobs | dbt dominates (thin AE sample) |
| Data scientist | ~19% (34) | ~1% (1) | 0 | Spark only in practice |
| Data analyst | ~2% (5) | ~2% (4) | 0 | Neither is a DA headline |
Among DE Spark jobs, ~59% also name Databricks; among DE dbt jobs, ~53% also name Databricks. So Databricks is a common platform wrapper for either skill, but listing Spark does not imply dbt (and vice versa).
What this means for your CV and interviews:
Postings rarely say "must know Databricks" in isolation. They ask for distributed data processing and modern warehouse tooling. Databricks bundles storage (Delta), compute (Spark), governance (Unity Catalog), SQL warehouses, and Jobs/scheduling in one product. Treating it as "only notebooks" undersells the platform.
| Stack you may see in NL | What it usually means | How Week 13 transfers |
|---|---|---|
| Databricks lakehouse | Delta + Spark + SQL warehouse + Jobs in one platform | Direct: this week's path |
| Snowflake or BigQuery (warehouse-first) | Strong SQL warehouse; Spark often optional or separate | dbt models and incremental thinking transfer; swap the adapter/profile |
| Microsoft Fabric | Microsoft-centric lakehouse / warehouse blend | Same lakehouse ideas; different UI and product names |
| Open lakehouse (Iceberg / Hudi + Spark or Trino) | Table format on object storage without full Databricks | Delta concepts transfer; engine and catalog differ |
| EMR / Synapse Spark / Dataproc | Cloud Spark without the full Databricks product | PySpark skills transfer; you still need a warehouse/orchestration story |
The practical implication: on your CV, frame Week 13 as "ported a production dbt project to a lakehouse platform, ran incremental models on 100M+ rows, and scheduled the build with a Git-backed Databricks Job", not "completed a Databricks tutorial". Recruiters care that you understand when scale tooling pays off, what stays the same (dbt models, tests, CI), and that the platform includes unattended runs.
Junior postings typically say:
Medior postings typically say:
Week 13 clears the junior bar: navigate a shared workspace, run PySpark safely, port dbt with only the profile changed, explain why the second incremental build is faster, and know that Workflows / Jobs is how the same platform schedules that work.
Strong CV line:
"Ported a Week 10 dbt mart (
fct_trips) from Postgres to Databricks on Delta Lake (128M rows), configured incrementalmergematerialization, and showed a full build vs incremental rerun that was substantially faster. Scheduled the same project as a Git-backed Databricks Job (manual green run, schedule paused on the shared workspace). Explored the dataset in PySpark with lazy evaluation and aggregation."
Recruiter keywords carried: Databricks, Delta Lake, dbt, PySpark, incremental models, lakehouse, Unity Catalog, Databricks Jobs / Workflows, Git-backed deploy.
Weaker alternative (avoid):
"Used Databricks and Spark in a course project."
When an interviewer asks "tell me about a time you worked with big data":
"Our analytics engineering project ran on Postgres until the data outgrew a single server. I changed only the dbt connection profile to target Databricks, kept the same models and tests, and switched
fct_tripsto incremental merge. The first build scanned the full history over 128 million rows; the incremental rerun finished much faster becauseis_incremental()limited the read to new rows. I scheduled that build as a Git-backed Databricks Job, proved one successful manual run, and paused the schedule on the shared bill. I also used PySpark for an exploration notebook and can explain when I would stay in dbt SQL instead."
Honest "what would you do differently?" follow-up:
"I would add job failure notifications and document the incremental boundary filter in the model YAML so the next person does not accidentally use
>=and create duplicates."
Percentages are measured from that crawl, not LinkedIn exports. Rounding and role tagging still leave room for noise, especially analytics engineer.
Generic NL junior career guidance (salary bands, day-to-day expectations, what employers do not expect from any junior) belongs on the shared cross-curriculum career page. That page does not exist yet; until it does, use this week's page only for Databricks / lakehouse signals.
Next: Assignment: Databricks Lab puts these skills in a portfolio-ready PR.
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.