Week 15 - Final Project

Project Ideas and Inspiration

Cross-Track Collaboration

The Project Architecture

Ingestion, ACA, and CI/CD

dbt models

Publish to Postgres

Running the Pipeline Locally

Orchestrating with Airflow

Promoting to Production

Secrets & Variables

Optional: Failure Alerting

Optional: Enriching with an LLM

Optional: The Health Page

Optional: Inbound Sync

Team Timelines

Gotchas & Pitfalls

Frequently Asked Questions

Glossary: Week 15

Career relevance: Week 15

Promoting to Production

Orchestrating with Airflow got Mode 1 and Mode 2 green on your laptop. This chapter is what happens after that on the team Airflow VM and in Azure: Mode 3 (platform-dev on main) and Mode 4 (prod).

A merge to main updates three things: the ingestion image, the prod Container Apps job, and the DAG on the team Airflow VM. Promoting is not a separate deploy step you run from your laptop. It is opening a pull request, getting it reviewed, and merging. What changes in production depends on which files your PR touched.

<aside> 📘 Recap from Week 5: merge-to-main CI is the same GitHub Actions pattern from Python CI pipeline. Here build and deploy-prod run only on merge, not on every pull request.

</aside>

The promotion ladder

Four Airflow targets (plus Path A with no Airflow). Each green run proves something the previous one could not. Landing and dbt schema are separate: Mode 2 and Mode 3 share aca-dev landing, but only Mode 3 writes the shared dev_airflow schema.

Mode 1 and Mode 2 set INGEST_MODE in airflow_settings.yaml (not .env) and restart Astro. Mode 2 also points .env LANDING_PATH at the aca-dev parent (team A: …/aca-dev/events). Mode 2 does not auto-run your uncommitted branch in ACA: only whatever image and job env Azure already has.

Mode 3 and Mode 4 both run on the team VM from main (git pull about every minute after merge). Both schedule at 09:00 Europe/Amsterdam. Mode 3 is created paused: unpause it once so the morning smoke runs. Mode 4 is the prod pipeline you monitor every weekday (optional Slack alerts: Failure Alerting). Your laptop cannot write Mode 4 schemas or the prod landing volume.

<aside> 💡 Mode 2 and Mode 3 share aca-dev landing and job-fp-ingest-dev, but Mode 2 still builds into your personal dev_<you> schema. Only Mode 3 uses the shared dev_airflow schema.

</aside>

How this chapter is structured

Same pattern as chapters 10 and 11: starter first, then your source.

Finish beat 1 before beat 2. Ingest and dbt code changes: stage chapters + ch10 beat 2. How data-ci-cd.yaml works: Ingestion, ACA, and CI/CD. Astro Mode 1/2 on that code: ch11.

By the end of this chapter, you should be able to:

• Trigger the production starter pipeline unchanged and confirm fresh rows in analytics
• Explain what runs on merge to main for ingestion code vs DAG-only vs dbt-only changes (detail in the ingestion chapter)
• Point both ACA jobs at your source via the workflow SOURCE_API_URL literal and verify job env after merge
• Update Airflow Variables on the team VM (parent LANDING_PATH / LANDING_PATH_DEV) so Mode 4 and Mode 3 match Mode 2 locally
• Verify the prod ACA job image tag matches your merge commit SHA
• Confirm prod Postgres (analytics) has fresh rows after a scheduled run
• Describe why build and deploy-prod skip on pull requests but run on merge

Run the prod baseline unchanged

Do this before your first feature pull request. You do not have to write any code to see the prod pipeline work. A starter version of every part has already been deployed for your team.

This section uses your team's prod Airflow VM in the browser, not Astro on your laptop. Mode 1 and Mode 2 in the previous chapter prove the DAG on your machine; Mode 4 proves the same DAG on the schedule that writes analytics. VM-only credentials (Airflow UI password, service principal, prod Postgres secret) are in Secrets & Variables.

Step 1: Open your team's prod Airflow UI from the table above.

Step 2: Sign in to the prod Airflow UI:

• Username: student-<your-name> (same <your-name> as in your Key Vault secret; see Airflow UI login)
• Password: fetch fp-airflow-password-<your-name> from Key Vault (not the Postgres password):

az keyvault secret show --vault-name kv-hyf-data \
  --name fp-airflow-password-<your-name> --query value -o tsv

Step 3: Find the final_project_pipeline DAG. Unpause it if needed, open Graph view, and click Trigger DAG.

Step 4: Wait until ingest → dbt_build → publish_to_backend are all green.

Prod Airflow overview for final_project_pipeline with recent successful runs and the task graph

<aside> 💡 Your DAG may show an extra enrich task if your team added it; the three core tasks above are what the starter baseline must green.

</aside>

Step 5: Run the same shape of query against the prod schema:

select count(*) as rows,
       count(distinct posting_id) as ids,
       max(ingested_at) as latest
from analytics.fct_postings;

<aside> 💡 Prod resource names differ from dev on purpose. The table in Running the Pipeline Locally is the map. Never point CLI commands or browser tabs at another team's URL by mistake.

</aside>

If the baseline run does not go green, or Postgres stays empty after a "successful" run, stop and tell a teacher which task failed and what the task log says. That is a platform issue, not a student exercise.

<aside> 💡 Recap: both baselines now exist. Dev runs on your laptop into analytics_dev, prod runs on your team's Airflow VM into analytics, and you have seen rows in each. From here every change you make is measured against these two known-good runs.

</aside>

What merge actually changes

Verified against the team repos' data-ci-cd.yaml workflow. Job-by-job walkthrough: data-ci-cd.yaml walkthrough.

Mode 4 runs the same code as your laptop, as different identities. What changes on promotion:

flowchart LR
    merge[Merge to main]
    build[CI build job]
    dprod[CI deploy-prod]
    ddev[CI deploy-dev]
    acr[ACR pipeline:sha]
    aca[job-fp-ingest]
    acadev[job-fp-ingest-dev]
    vm[Airflow VM git pull]
    dag[final_project_pipeline]

    merge --> build --> acr
    acr --> dprod --> aca
    acr --> ddev --> acadev
    merge --> vm --> dag

<aside> 💡 DAG green before your custom dbt is done is fine. Mode 1 on the starter pipeline proves orchestration; you can swap models in later without blocking the chain.

</aside>

Keep the merge diagram above next to you while you promote: it is the map for everything in this chapter.

<aside> 💡 Editable deep-dive template (optional): week_15__architecture.excalidraw. Prefer a mermaid diagram in your team README.

</aside>

From starter to your source (where each chapter fits)