Week 15 - Final Project

Project Ideas and Inspiration

Cross-Track Collaboration

The Project Architecture

Ingestion, ACA, and CI/CD

dbt models

Publish to Postgres

Running the Pipeline Locally

Orchestrating with Airflow

Promoting to Production

Secrets & Variables

Optional: Failure Alerting

Optional: Enriching with an LLM

Optional: The Health Page

Optional: Inbound Sync

Team Timelines

Gotchas & Pitfalls

Frequently Asked Questions

Glossary: Week 15

Career relevance: Week 15

Ingestion, ACA, and CI/CD

The final project pipeline has three stages: ingest, dbt, then publish. This chapter is only the first stage - and it is three layers students often treat as one:

1. The Python package under data/src/ingestion
2. The Container Apps jobs that run that package in Azure (job-fp-ingest-dev and job-fp-ingest)
3. The GitHub Actions workflow that builds the image and rewrites job env on every merge to main

Read this before you change a feed or open a Mode 2 spike. You do not need to merge or start a job yet. Running the Pipeline Locally is where you run Path A by hand. Orchestrating with Airflow and Promoting to Production are where you use ACA and CI after Path A is green.

Why this order matters: the platform shape (why ACA exists, Managed Identity) is in The Project Architecture. This page is the file map and delivery path for ingestion only.

By the end of this chapter, you should be able to:

• Name which files own fetch, Pydantic validation, and the landing folder constant
• Explain where laptop SOURCE_API_URL applies vs where ACA job env applies
• List the jobs in data-ci-cd.yaml and which run on a PR vs on merge to main
• Say what deploy-prod and deploy-dev write onto each Container Apps job
• Know not to invent a second ACA job when you swap APIs

The three layers

flowchart TB
    subgraph code["1. Python package"]
        pkg["data/src/ingestion<br/>fetch · validate · write JSON"]
    end
    subgraph aca["2. Container Apps jobs"]
        devj["job-fp-ingest-dev<br/>dev / aca-dev"]
        prodj["job-fp-ingest<br/>prod / raw"]
    end
    subgraph ci["3. CI on merge to main"]
        wf["data-ci-cd.yaml<br/>build · deploy-dev · deploy-prod"]
    end
    pkg -->|image pipeline:sha| aca
    wf -->|push image + set job env| aca

Airflow (Astro or the team VM) only starts a job. It does not pass your laptop .env into the container. The container always uses job env in Azure plus the code baked into the image tag the job points at.

<aside> 💡 Path A and Astro Mode 1 skip layers 2 and 3: local Python reads data/.env and writes under your personal prefix. Mode 2 and Modes 3–4 use layers 2–3. Same package, different runners.

</aside>

Layer 1: the Python package

Keep the package layout. Rewrite the API boundary and the Pydantic shape. Do not invent a second Container Apps job when the feed changes.

Ingestion stays thin: call the API, validate with Pydantic (Week 3), write JSON to the landing path, stop. Cleaning, deduplication, and joins belong in dbt (dbt models).

Raw vs validated. Put source-specific cleaning in parse_records / Pydantic when a bad row must not land. Put type casts and renames in staging so Catalog Explorer still shows inspectable JSON.

Prefer keeping the landing folder name postings (teams B/C) or events (team A) even when the API changes. Renaming the folder forces updates to pipeline.py, staging SQL, and every teammate's paths together.

https://gist.githack.com/lassebenni/acaa5f82c778b2b5974e28cbabefaf59/raw/week_15__ingest_package_code_explainer.html

Hover the blue lines for fetch, Pydantic validation, SOURCE_NAME, and how load_config reads the process environment.

<aside> 📘 Recap from Week 3: validate API response shape with Pydantic before you land JSON. See Pydantic validation. Skipping validation because one sample "looks fine" is how bad rows reach staging.

</aside>

Laptop env vs landing write path

On Path A and Astro Mode 1, local Python reads data/.env:

The landing subfolder (postings / events) is not an .env key. It is SOURCE_NAME in pipeline.py. Full lookup tables: Secrets & Variables.

<aside> ⌨️ Hands on: Open data/src/ingestion/pipeline.py in your team monorepo and find SOURCE_NAME. Note whether your team uses postings or events. Do not change it yet.

</aside>

Layer 2: Container Apps jobs

Your team has two ingestion jobs. Same image family; different landing targets:

You do not create a new job when you swap APIs. Merge (or a careful hand az containerapp job update while spiking) updates image and env on these two names.

Job env that matters for the feed:

Airflow Variables such as ACA_INGEST_JOB only name which job to start. They do not carry SOURCE_API_URL. Details: Week 6: Container Apps Jobs and Architecture: why ACA.

https://gist.githack.com/lassebenni/af1d21f39b004ab9f858f4facdf2766c/raw/week_15__aca_start_wait_code_explainer.html

Airflow POSTs start, then polls until the execution finishes, so dbt does not run on empty landing.

<aside> ⚠️ Editing only ingest.py and local .env leaves Mode 2 and prod on the starter URL until workflow literals change and deploy-dev / deploy-prod succeed.

</aside>

Layer 3: how data-ci-cd.yaml works

Open .github/workflows/data-ci-cd.yaml in your monorepo. Example (team A): data-ci-cd.yaml on main. Swap the group letter for B or C.

Team Azure IDs and SOURCE_API_URL live as literals in the workflow top-level env: block, not as GitHub Actions repository Variables. A PR can change the URL in one reviewable diff. Both deploy jobs read ${{ env.SOURCE_API_URL }} - edit the top-level SOURCE_API_URL: once; do not paste the URL into three places.

There is no registry password in the workflow. GitHub exchanges a short-lived OIDC token for Azure access (same idea as Week 5 CI and Architecture: ACR). Trust is pinned to main of your repository, so a pull request cannot deploy.

Which jobs run when

Path filters: the workflow runs when data/** or the workflow file itself changes. workflow_dispatch lets teachers rebuild without a code change.

flowchart LR
    merge[Merge to main]
    build[build]
    acr[ACR pipeline:sha]
    dprod[deploy-prod]
    ddev[deploy-dev]
    jprod[job-fp-ingest]
    jdev[job-fp-ingest-dev]
    vm[Airflow VM git pull]

    merge --> build --> acr
    acr --> dprod --> jprod
    acr --> ddev --> jdev
    merge --> vm

DAG and dbt code on the VM arrive via git pull, not via this workflow. The workflow owns the ingestion image and ACA job env only.

<aside> 🤓 Curious Geek: why ACR, not GHCR, for the pipeline image?

</aside>

What each deploy writes

deploy-prod sets (among other vars) LANDING_CONTAINER=prod, LANDING_PREFIX=raw, and SOURCE_API_URL from the workflow env:.

deploy-dev sets LANDING_CONTAINER=dev, LANDING_PREFIX=aca-dev, and the same SOURCE_API_URL.

CI tags both pipeline:<commit-sha> and pipeline:latest, but deploy always points the job at the SHA. A job running latest was pointed by hand, not by CI.

API keys are different: never commit them. Add a GitHub Actions secret, wire secrets.YOUR_KEY into the deploy job, and set the ACA env with secretref:… in the workflow. See Secrets & Variables.

https://gist.githack.com/lassebenni/9753a184333d4688fb4c4d6ac5f280aa/raw/week_15__data_ci_cd_deploy_code_explainer.html

Trace the top-level SOURCE_API_URL literal into both deploy jobs' env_vars. That is what the container sees, not your laptop .env.

<aside> ⌨️ Hands on: In your clone, open data-ci-cd.yaml and find (1) top-level SOURCE_API_URL, (2) the deploy-prod and deploy-dev job names, (3) where ${{ env.SOURCE_API_URL }} is passed into env_vars. Do not merge a change yet.

</aside>

When you swap the source (order of operations)

1. Prove a local fetch, then Path A on the starter (ch10).
2. Change the Python package + dbt (+ publish if needed) and re-run Path A on your feed.
3. Prove Astro Mode 1 (and Mode 2 only after the dev job image/env match what you need - Mode 2 starts whatever Azure already has).
4. In the same PR as ingest: set top-level SOURCE_API_URL in data-ci-cd.yaml; refresh .env.example.
5. Merge; wait until build, deploy-prod, and deploy-dev are green.
6. Verify job env and (for prod) image SHA as in Promoting to Production.

Until step 5 succeeds, Mode 2 and Mode 4 can still hit Arbeitnow even when your laptop .env names FreeHire or Ticketmaster.

Ready for the next chapter when

dbt models assumes you know where JSON lands and that ACA/CI are how that same package ships outside your laptop.

• [ ] You can name the ingest files that own fetch, models, and SOURCE_NAME
• [ ] You can explain why Mode 2 ignores laptop SOURCE_API_URL
• [ ] You can list which data-ci-cd.yaml jobs run on a PR vs on merge
• [ ] You know both ACA jobs are updated by CI - you do not create a third job for a new API

Extra reading

• Week 5: Azure Container Registry and Python CI pipeline