Optional: Enriching with an LLM
This chapter is the checklist. It says which projects qualify, what your team hands in at the end of week 5, what you personally are responsible for, and how the demo runs. Everything else in Week 15 explains how to build it. This page says what has to exist.
By the end of this chapter, you should be able to test a project idea against both requirements, list what your team delivers, and say what you are individually responsible for.
The six suggested projects all pass both tests. If you would rather build something else, propose it, and hold it against these two before you do. A teacher approves the final choice in week 1.
The app must store something users create. The backend and frontend need that state for the product to be real. Your data pipeline starts with an external source. The reasoning is in Project Ideas and Inspiration, so here it is as one question: if the backend trainees shipped nothing at all, would there still be a product? If yes, the idea is too thin.
The final project is where the whole track comes together, so an idea only qualifies if it needs every piece of it. Answer yes to each line:
<aside> 📘 Recap from Weeks 5–13: this checklist is skills you already have: containers and CI in Week 5; blob, jobs, and Key Vault in Week 6; dbt in Week 10; Airflow in Week 12; Databricks in Week 13. The final project is where you use them together on a problem nobody solved for you.
</aside>
Optional after prod is green end to end (nice to have, not required for the demo): see Optional: Inbound Sync.
If your idea cannot tick one of those lines, the problem is not the idea: it is that half the track would go undemonstrated. Change the idea, or change how you get the data.
<aside> 💡 Failure alerting and a pipeline health page are worth building and are not required. The Project Architecture lists every component, including which ones are marked optional.
</aside>
One repository and one demo. Nothing is submitted separately.
The pipeline, running end to end on a schedule without anyone pressing a button. A green run that only happened because somebody triggered it ten minutes before the demo does not count.
Outbound sync. Publishes your marts into the schema the backend reads. Required. Ownership is in Cross-Track Collaboration; implementation checklist in Optional: Inbound Sync.
Tests, at minimum on the mart the backend reads. A published table with no tests on its keys is the most common way a demo breaks in front of people.
Monitoring (optional). A Slack failure alert and a pipeline health page help when something breaks overnight, but neither is required for the demo. See Optional: Failure Alerting and Optional: The Health Page.
The README, holding four things: the architecture diagram from week 1, a data dictionary for every published mart, the source reports, and a short section per data trainee naming the tools they own.
<aside> ⚠️ The architecture diagram is not a week 5 deliverable you recreate from memory. Draw it in week 1 and edit it whenever something changes. By the demo it should have been revised several times.
</aside>
Your team's result is not your result. Each data trainee is responsible for the items below.
Starter pipeline green before feature work. Read Ingestion, dbt, and Publish, then complete Running the Pipeline Locally (Path A by hand), Orchestrating with Airflow (Mode 1), and trigger the prod baseline once in Promoting to Production on the starter feed before opening pull requests that change ingestion, dbt, or publish. That is how you know the platform works before you touch it.
One source together, then dev, then prod. All DEs compare feeds and pick one. Share a real sample with backend before dbt work. Pair on local Python ingestion, then green the dev pipeline on that source, then prod. Only then split tools or add a second source. See Team Timelines.
Real source samples only. Hand the backend real JSON from your chosen source in week 1. Do not generate fake source data because the feed is hard to find.
Two tools, not one layer. You own one tool in weeks 1 to 3, then a different one from week 4 after the handover. Which ones depends on your team, and Team Timelines has the table. Both are yours to explain.
A clean handover. At the end of week 3, after the first source is prod-green, show your teammate how your tool runs, what is weak about it, and a question you never had time to answer. Then write that up in the README.
Being able to explain a part you did not build. This is the whole point of the tool switch, and it is checked at the demo rather than assumed.
Ten minutes, then questions. Structure it like this:
The problem, in one minute. What the product does, and who would use it.
The architecture, in two minutes, pointing at your diagram. Where the data comes from, where it lands, how it reaches the screen.
A live run, in five minutes. Show the pipeline running, or show the run that happened on its schedule this morning. Then show the result in the application itself, not in a SQL client.
What broke, in two minutes. The failure you did not expect, and what you changed. This is the part interviewers remember.
<aside> 💡 Present a part you inherited at the handover rather than one you built from scratch. If you can explain your teammate's tool, the switch worked. Practise it in week 5 before you have to do it in front of teachers.
</aside>
You will run out of time. That is expected. Cut in this order, and say so in your README rather than hiding it:
Never cut the outbound sync, the tests on the mart the backend reads, or the daily schedule. Those three are what make this a data product rather than a script.
Teachers look at these, roughly in this order:
| Dimension | What is looked at |
|---|---|
| Ingestion | Pages, request limits, retries, loading only new data, and what happens when the source changes shape |
| Modelling | Staging and mart layers, a grain (what one row represents) you can state in one sentence, tests that would actually catch something |
| Platform | Deployed on Azure, secrets in Key Vault or a secret scope, nothing running from a laptop |
| Orchestration | A real schedule, dependencies between tasks, and retries (Orchestrating with Airflow). Slack alerting is optional |
| Outbound sync | Published in one step, safe to re-run, fails loudly rather than quietly |
| Inbound sync (optional, the reverse copy: application tables into your warehouse) | If built: read-only role, no PII, documented in README |
| Documentation | The README, the diagram, the data dictionary, the source reports |
| Collaboration | Pull requests reviewed by a person, and whether you can explain a part you did not build |
Two things are checked before anything else, and getting them wrong caps the result however good the rest is:
Nothing is built yet, so there is no state to check. These are recall checks, and the next chapter assumes you already know what you are aiming at.
Next up: Cross-Track Collaboration, where you learn how the three tracks divide the work, why your pipeline never writes into the application's own tables, and what the mart contract must contain. Outbound sync implementation details are in Optional: Inbound Sync.