Optional: Enriching with an LLM
Your final project is built by a cross-track team: two or three Data Track trainees, two Backend Track trainees, and one Frontend Track trainee. This page gives you six vetted project options to choose from, and explains what each team member owns.
By the end of this chapter, you should be able to pick a project whose data source you have verified works, and name which part of it you personally own.
The product must have user-generated state the backend and frontend use. Your data pipeline starts with an external source you fetch yourself.
That sounds abstract, so here is what it means in practice. If your team picks a public API and simply displays it, the backend trainees have nothing to build except a pass-through, and the frontend trainee is left with a chart page. Nobody's skills get used.
Test every idea with this question: if the backend pair shipped nothing at all, would there still be a product? If the answer is yes, the idea is too thin. Pick another.
All six options below pass that test.
You are not limited to them. Propose your own project if you would rather build something else: it has to pass this test and the technology test in Requirements and Deliverables, and a teacher approves it in week 1.
Before you commit to a project, all data trainees compare candidate feeds together. Spend a day per option: can you get it legally, what is in one record, does it answer the product question? Put the comparison in the repo (a short table is enough), then agree on one source for the first end-to-end chain.
Write a full source report only for the source you chose. Scout notes on runners-up can be one paragraph each.
<aside> 📘 Recap from Week 3: validate API response shape with Pydantic before you land JSON. See Pydantic validation. The final project uses the same pattern in ingestion; do not skip validation because the source "looks fine" in one sample file.
</aside>
After you pick a source, download a real extraction (one API page, one scrape run, one file). Commit or attach the sample and a field list. Hand that to backend and frontend before you write dbt models or open container pull requests. They stub endpoints from your file while you build ingestion.
Then pair on a local Python script that reproduces the fetch. Only when that script works on your laptop do you wire the dev pipeline (Running the Pipeline Locally), orchestrate it (Orchestrating with Airflow), and promote to prod (Promoting to Production). The full sequence is in Team Timelines.
requests call from that. You are calling their API, not parsing HTML. Respect terms of service and rate limits.<aside> ⚠️ Legal and practical limits: read the site's terms. Stay within rate limits. Do not rotate credentials or use proxies to evade blocks. "Educational use" does not override a site's rules.
</aside>
Sources often block cloud IP ranges. If ingestion succeeds locally and fails from Container Apps with 403 or empty responses, suspect blocking before you debug Python. Prefer an API that allows server-side access.
Commit one page for your chosen source to the repo (plus a short comparison table if you evaluated alternatives):
az containerapp job), not only your laptopUse this when there is no public API documentation.
requests and inspect the JSON shape.<aside>
⌨️ Hands on: Pick any page that loads list data via XHR (not the job-board starter). Copy one request as cURL, run it with curl in your terminal, and save the JSON. That file is your first deliverable to the backend team.
</aside>
Only after dev and prod pipelines work for your chosen feed:
Each track owns one layer. Your data marts do not go straight to the browser: the backend serves them, which is exactly how it works in a real company. See the cross-track diagram in Cross-Track Collaboration.
<aside> ⚠️ Your Metabase and Streamlit work from Week 11 is for pipeline operations only: freshness, row counts, and test results. It is not the user-facing surface. Building a second student-facing dashboard leaves your frontend teammate with nothing to do.
</aside>
Five or six people, each with a distinct area of ownership. Agree on these in your first team meeting.
Weeks 1 to 3: all data trainees work together on source choice, the sample handoff, local ingestion, and the first dev and prod pipeline. From week 4: the data seats own tools and switch once, as Team Timelines sets out.
| Seat | Owns (from week 4) | Builds on | What breaks in the demo without it |
|---|---|---|---|
| Data 1 | Ingestion, then orchestration and outbound sync, or data quality in a 3-DE team | Weeks 3, 6, 7, 10, 12 | No fresh data. You demo a stale snapshot with no schedule. |
| Data 2 | The platform, then models, marts and classification | Weeks 6, 10, 13, 14 | Endpoints return raw rows. No tests, so nobody can trust the numbers. |
| Data 3 (3-DE teams) | The second source after the first is prod-green, then orchestration and outbound sync | Weeks 4+ | Nothing runs on a schedule, or the mart lacks the second feed. |
| Backend 1 | Domain model and authentication | Backend weeks 4, 5, 6, 10, 11 | No login and no user data. The app is a public read-only page. |
| Backend 2 | Events, serving, observability | Backend weeks 9, 12, 13, 14 | Pipeline and app never talk. No logs when the demo breaks live. |
| Frontend | The product surface | Frontend weeks 11, 12, 13, 14 | There is no demo. |
<aside> 💡 With only one frontend trainee, every demo runs through them. Agree on roughly three screens done well rather than ten screens half-built.
</aside>
Each option uses the same architecture. What differs is the hard engineering problem at its centre, so three teams picking three different options will not build the same thing.
Trainees track demand for roles and skills. Users save searches, bookmark postings, and log what happened when they applied. The analytics view shows which skills are rising, and which ones actually lead to interviews.
User-generated state: saved searches, bookmarks, application outcomes.
Hard part: using an LLM to classify free-text postings into a role and skill taxonomy, and coping when that taxonomy drifts over time.
<aside> 💡 Editable template for Project A (optional): week_15__job_app_example.excalidraw. Teams on other projects use the generic template linked from Cross-Track Collaboration. Prefer a mermaid diagram in the team README.
</aside>
Commuters register the routes they travel regularly. The app reports personal reliability, for example "your 08:12 is late 40% of the time", and lets users submit disruption reports that the pipeline joins against official data.
User-generated state: registered routes, crowd-sourced disruption reports.
Hard part: data freshness with a real service-level target, and event-driven ingestion where reports arrive as messages rather than in a nightly batch.
Contributors register the repositories they work on and set personal goals. The app pulls their activity and reports progress, streaks, and how fast their reviews come back.
User-generated state: registered repositories, personal goals.
Hard part: per-user incremental fetching against strict rate limits, which makes parameterized runs and backfill genuinely necessary rather than decorative.
Residents subscribe to their postcode and get alerted when a building or environmental permit is filed nearby. They can record objections and see which application types draw the most opposition.
User-generated state: postcode subscriptions, objections, notification preferences.
Hard part: geographic joins, plus extracting structure from semi-structured Dutch notice text.
Households register their profile and log appliance usage. The app advises the cheapest windows to run the washing machine or charge a car, and reports what they saved.
User-generated state: household profiles, appliance logs, uploaded meter readings.
Hard part: time-series cost simulation, and handling data that is genuinely personal, which makes the Backend Track's sensitive-data lessons load-bearing.