Optional: Enriching with an LLM
This chapter is optional. Nothing here is required for Week 15's deliverables or the assignment. The failure alert from Getting Told When It Breaks is the required monitoring, and it stays required whether or not you build this page.
A health page answers a question the alert cannot: not "did something break?" but "is the data any good right now?". Those come apart more often than you would expect.
By the end of this chapter, you should be able to:
Airflow already tells you whether the DAG went green. It cannot tell you whether the data is any good, because a task that fetches zero records still succeeds.
Picture the grid on a morning when the source API returned an empty list. Three green squares, no alert, nothing to investigate. The warehouse is empty and the app is showing nothing. Everything worked exactly as instructed, which is the problem: an empty warehouse behind a green grid is the failure nobody investigates.
So the rule for every panel on this page:
<aside> ❗ Read the data, not the job. Count rows and list files. Show the task status underneath as context, never as the verdict.
</aside>
That inverts the usual instinct, which is to build a page of task statuses. Task statuses are already in Airflow, and you can already see them there.
Start with what you would check at 09:00 on a morning when a teammate says the site looks wrong. That is usually three things:
| Panel | The question it answers |
|---|---|
| Freshness | When did data last arrive, and is that within a day? |
| Volume | How many rows are there, and is that roughly yesterday's number? |
| Quality | Did the dbt tests pass on the last run? |
optional/streamlit/app.py in your repo ships the first two. It reads the backend database with the same BACKEND_PG_* settings the sync uses, deliberately: a dashboard with its own connection settings is a dashboard that quietly points at the wrong server for a week.
It is about seventy lines and it is meant to be extended. The caption in it says so.
<aside> ⚠️ This page reads the end of the pipeline, the published mart. That means it cannot distinguish "ingestion failed" from "dbt failed" from "sync failed". It tells you the product is stale. Finding out which step is Airflow's job, and the alert already links you there.
</aside>
The third panel is the one you cannot build yet, and the reason is worth understanding.
dbt writes a full account of every model and test it ran to target/run_results.json. Then that file sits on the machine that ran it. So "are the tests passing?" can only be answered by whoever has access to that VM and a reason to look, which is nobody at 09:00.
optional/dbt_results/ fixes that by parsing the file after each run and writing one row per node into <catalog>.ops.dbt_test_runs. Test outcomes become data: queryable with SQL, joinable to anything, readable by a page that has no idea Airflow exists.
Two details in that code are worth reading rather than copying:
It publishes before the task decides its fate. A failing test is exactly the one you want recorded, so the write happens before the exit code is checked. Record first, then fail.
Nothing in it raises. dbt's own exit code already decides whether the pipeline failed. Losing the bookkeeping is annoying; failing a green run because the bookkeeping stumbled would be worse.
<aside>
⚠️ The ops schema is owned by the scheduled run. Your own account can read it and not write it, so this runs in Airflow, not from your machine. Trying it locally gives you PERMISSION_DENIED, which is the dev/prod split working rather than something misconfigured.
</aside>
The two halves are shipped separately and are not joined up for you. The page queries Postgres; pointing a panel at ops.dbt_test_runs in Databricks is the piece you would write. That pair is what turns the page from "did anything arrive?" into something worth opening.
Not on your laptop. In Week 11 you ran Streamlit locally, which is right for learning and wrong for a page your team is meant to check. A page that only exists while one person's laptop is open is that person's page.
Deploy it as a Container App next to your pipeline, using the same az acr build and az containerapp path your ingestion job already uses. Set --min-replicas 0: nobody watches this continuously, so accept a few seconds of cold start and pay nothing while it sleeps.
The page's identity needs read access to whatever the panels read, and nothing more. A page that can write to your mart is a page that can corrupt it.
<aside> ⚠️ Ingress makes the URL public and unauthenticated. This page shows row counts and timestamps, not personal data, so that is an acceptable trade. Make it a decision your team states in the README, not an accident somebody discovers.
</aside>
Whatever your page claims, verify one number independently before you trust it.
Step 1: Read the row count off the page.
Step 2: Get the same number straight from the database:
select count(*) from analytics.fct_postings;
Step 3: If they disagree, the page is caching. @st.cache_data(ttl=60) means it can be a minute stale, which is fine as long as you know it.
A dashboard nobody has checked against reality is decoration. This takes two minutes and is the difference between a panel you believe at 09:00 and one you argue with.
This chapter is optional, so the check is whether you made a decision rather than whether you built something.
Next up: Team Timelines, where you find the week-by-week plan for your team size and seat.
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 https://hackyourfuture.net/
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.