Optional: Enriching with an LLM
Plain-language definitions for capstone vocabulary. Entries follow the chapter that first introduces the term in the Week 15 reading order.
A one-page document per data source covering legal access, volume, freshness, fields in one record, and whether the source answers the product question. Committed to the repo in week 1.
Used in Final Project Brief
The written agreement between data and backend on a published table: grain, columns, refresh cadence, and nullable fields. The backend builds API endpoints against this shape.
Used in Final Project Brief
Running ingest, dbt, and publish by hand in order before orchestrating through Airflow. Proves the chain works before you change starter code.
Used in Final Project Brief
Astro on your laptop with INGEST_MODE=local: ingest, dbt, and publish all run in the Astro worker from your branch. Ingest writes under your personal landing prefix; dbt builds into dev_<yourname>; publish lands in Postgres analytics_dev.
Used in Orchestrating with Airflow
The level of detail one row represents, stated in one sentence (for example one row per posting per day). Wrong grain breaks joins and API endpoints.
Used in Requirements and Deliverables
The data-to-backend copy: your marts leave Databricks/Postgres publish step and land in the analytics schema the backend reads.
Used in Cross-Track Collaboration
The backend-to-data copy of selected application tables into your warehouse for models that need user behaviour. Optional phase 2 only. See Optional: Inbound Sync.
Used in Cross-Track Collaboration
The agreement on which app-generated events or tables the data team may ingest later, how often, and which fields must never include PII. Outline in week 1; full architecture in Optional: Inbound Sync.
Used in Cross-Track Collaboration and Optional: Inbound Sync
A published consumption table with one row per entity (for example one row per job), with attributes such as company name and location already on the row. This is the One Big Table (OBT) pattern. Built from staging and optional int_* models. This is what the backend reads. See Week 10 Materializations.
Used in Cross-Track Collaboration
A published table at a different grain from the wide mart (for example one row per job × skill). Name it after the relationship (job_skills). Publish it only when the backend needs that access pattern.
Used in Cross-Track Collaboration
A pre-aggregated published table at a stated grain (for example top skills per role). Optional alternative to backend scheduled caches for KPIs the product shows often.
Used in Cross-Track Collaboration
The Unity Catalog path where raw JSON files land after ingestion, one file per run, before dbt reads them.
Used in The Project Architecture
The smallest end-to-end path that proves data flows on your first chosen source: local fetch, one ingest, one staging model, one mart, one publish, dev pipeline green, then prod. Built by all DEs together in weeks 1 to 3 before the tool split.
Used in Team Timelines
Astro on your laptop with INGEST_MODE=aca: ingest starts job-fp-ingest-dev (shared aca-dev landing); dbt and publish still run from your branch into dev_<yourname> and Postgres analytics_dev. Proves the Container Apps image and job env before you merge.
Used in Orchestrating with Airflow
Publishing pattern: load new mart rows into a staging table, then swap or rename so the backend sees a consistent snapshot atomically.
Used in Optional: Inbound Sync
Team Airflow VM DAG final_project_pipeline_dev: same shape as Mode 2 (dev ACA job, aca-dev landing, publish to analytics_dev) but code comes from main via VM git pull, and dbt writes the shared Databricks schema dev_airflow. Scheduled daily at 09:00 Europe/Amsterdam (unpause once). Last check that the full path works on main before trusting Mode 4.
Used in Promoting to Production
Team Airflow VM DAG final_project_pipeline (prod): job-fp-ingest, prod landing, Databricks analytics, Postgres analytics. Same schedule (09:00 Europe/Amsterdam). Your laptop cannot write these paths; only merge + this DAG can.
Used in Promoting to Production
The end-of-week-3 session where each trainee demos their tool, names one weakness, and leaves a README section for the person who inherits it in week 4.
Used in Team Timelines
Every data/.env setting, Airflow Variable, and Key Vault secret for the final project lives in Secrets & Variables (reference lookup, grouped by the pipeline chapter where you first need them).
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 https://hackyourfuture.net/
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.