Optional: Enriching with an LLM
Running ingestion, dbt, and publish by hand proves your code logic. Running through Airflow proves orchestration logic: task order, container environment, and scheduler behaviour. Resource names follow the team-a example from Running the Pipeline Locally; substitute b or c on other teams. Astro-specific .env keys are in Secrets & Variables. Production is Promoting to Production.
Same pattern as chapters 10 and 12: starter first, then your source.
| Beat | What you do | Jump to |
|---|---|---|
| 1. Starter | Mode 1 then Mode 2 on the template pipeline (Arbeitnow image/job env is fine). | One-time Astro setup → Mode 1 → Mode 2 |
| 2. Your source | After Path A on your feed is green, re-run Mode 1 and Mode 2 on that code. | Your source on Astro |
Finish beat 1 before beat 2. ACA job URLs and VM Variables for a lasting swap are ch12, not this chapter.
| Goal | When |
|---|---|
| Beat 1 (starter) | After Path A starter is green; week 1 orchestration proof |
| Beat 2 (your source) | After Path A beat 2 is green on your feed |
By the end of this chapter, you should be able to:
INGEST_MODE=local and read task logsAirflow validates things the script-only flow cannot:
ingest → dbt_build → publish_to_backend)In short: scripts prove each component works; Airflow proves the full pipeline is runnable the way it will run in operations.
| Task | Code | Who actually runs it |
|---|---|---|
ingest |
src/ingestion/pipeline.py in the pipeline image |
With INGEST_MODE=aca, Airflow starts the Container Apps job; the container executes the code. With INGEST_MODE=local, the worker runs Python directly. |
dbt_build |
data/dbt project, mounted into the worker |
Airflow runs dbt build as a subprocess; Databricks executes the SQL. |
publish_to_backend |
src/publishing/sync.py, mounted into the worker |
Airflow runs the Python script as a subprocess in the worker. |
dbt build does not need its own container image because the worker only compiles SQL and submits it to your warehouse. ingest does, because it must run on a schedule even when your laptop is off, with its own dependency set isolated from Airflow.
Mode 1 runs all three tasks inside the Astro worker. Mode 2 keeps ingest on a Container Apps job and leaves dbt_build and publish_to_backend on the worker. Modes 3 and 4 on the team VM also use ACA ingest (same idea as Mode 2); they differ by code source (main) and which landing / schemas they write.
You need Path A green from Running the Pipeline Locally: ingest, dbt, and publish by hand, with rows in Railway analytics_dev.fct_postings. For beat 1 that is the starter feed; for beat 2 it is your feed. You also need:
data/.env filled in (same file Path A used, including BACKEND_PG_* from fp-pg-host-team-x and the other Railway secrets; Astro mounts it into the worker containers)AZURE_CLIENT_ID and AZURE_CLIENT_SECRET in data/.env (see one-time setup below; required before your first Astro run)Path A on your laptop used az login for blob writes. Astro runs ingest inside Docker, where there is no interactive login. Add the dev-run service principal to data/.env before your first astro dev restart. The same principal uploads to the dev container in Mode 1 and starts job-fp-ingest-dev in Mode 2. Keep both vars in .env when you switch modes.
Fetch fp-devrun-client-id-team-x and fp-devrun-client-secret-team-x from the Orchestrating with Airflow matrix and set AZURE_CLIENT_ID and AZURE_CLIENT_SECRET (team-a names below):
az keyvault secret show --vault-name kv-hyf-data \
--name fp-devrun-client-id-team-a --query value -o tsv
az keyvault secret show --vault-name kv-hyf-data \
--name fp-devrun-client-secret-team-a --query value -o tsv
This is not the team's Databricks service principal. Do not use the Databricks SP secrets on your laptop.
Copy the settings template once so local Airflow defaults to INGEST_MODE=local:
cd data/airflow
cp airflow_settings.yaml.example airflow_settings.yaml
astro dev restart reads this file on every start. Chapter 10 does not need it; only Astro does.
.env vs airflow_settings.yaml| File | What goes here locally | In Admin → Variables? |
|---|---|---|
data/.env |
Almost everything: Databricks, Azure, Postgres, LANDING_PATH, tokens, dev-run SP |
No; Astro mounts this into worker containers |
data/airflow/airflow_settings.yaml |
INGEST_MODE only (local or aca); room for Connections/Pools later |
Yes; Astro re-seeds listed Variables on every astro dev restart |
Path A (uv run, preflight) and Astro share data/.env. That is why credentials and resource names stay there instead of the yaml: one file for scripts and DAG tasks alike. Secrets in the DAG use the environment (then Key Vault on the VM), not Airflow Variables; copying the prod VM Variable list into yaml would not help and would duplicate .env.
INGEST_MODE alone belongs in yaml because it is an orchestration toggle (like prod), not a credential; it is not in .env.example; and Astro re-applies yaml on every restart so Mode 1 stays the default after cp …example.
Skip the yaml copy → Mode 2 by default. If you never copy airflow_settings.yaml.example, nothing seeds INGEST_MODE=local. The DAG then falls back to aca: ingest tries to start a Container Apps job and needs extra Azure settings you have not wired yet. Copy the template before your first Astro run.
<aside>
💡 One Variable in the UI is normal. After restart you usually see only INGEST_MODE. For every other name in Secrets & Variables, check .env first; those keys are not Airflow Variables locally. To switch modes, edit variable_value in yaml and astro dev restart; a Variable seeded from yaml overrides the same name in .env if both exist.
Prod VM is different: most settings are Airflow Variables at provisioning (Promoting to Production). Do not mirror that list into local yaml; fill .env instead.
</aside>
Use this mode first. The DAG orchestrates all three tasks, but ingest runs local Python inside the Airflow worker instead of triggering ACA.
cd data/airflow
# Forces a full recreate so code, env vars, and airflow_settings.yaml reload.
astro dev restart
Use the URL astro dev restart prints, not a URL from an old session. See Gotchas if the port looks wrong.
astro dev restart seeds INGEST_MODE=local from airflow_settings.yaml (see one-time setup above).
The Astro UI is the same one you used in Week 12: Airflow fundamentals. Your DAG is final_project_pipeline with three tasks (ingest → dbt_build → publish_to_backend), not hello_pipeline with one.
Clicking the play button to trigger a DAG in Astro (Week 12)
final_project_pipelineingest → dbt_build → publish_to_backend)When the run finishes, the grid should look like Week 12's green column: every task cell dark green:
Astro grid view after a successful DAG run (Week 12 hello_pipeline example)
Task instance logs in Astro (Week 12 example)
Completed successfullypublished ... rowsselect count(*) as rows,
count(distinct posting_id) as ids,
max(ingested_at) as latest
from analytics_dev.fct_postings;
Mode 1 exit criteria: DAG run succeeds with INGEST_MODE=local, and Railway Postgres shows fresh rows in analytics_dev.fct_postings (same host as BACKEND_PG_HOST / fp-pg-host-team-x from Key Vault, not a local container).
Mode 2 starts the dev Container Apps job from Astro instead of local Python. After Mode 1 is green, switch the same DAG to ACA-backed ingest for full integration validation. Astro is already running. You only need one more restart after editing .env and airflow_settings.yaml.
<aside> 💡 DE2 platform seat, week 3+. Every data trainee should read this section. Only the platform owner needs to run the image build and job update commands.
</aside>
Mode 2 does not build or deploy on merge to your branch. The ingest task starts an existing Container Apps job and waits for it (POST …/start with an empty body). Astro does not pass your laptop SOURCE_API_URL (or other ingest env) into that start. The container always uses job env in Azure. Why, and how CI rewrites that env: Ingestion, ACA, and CI/CD. Build, push, and point the dev job at your image by hand first, or a green ingest task only proves the old image still works. A green Mode 2 run with a FreeHire .env can still be fetching Arbeitnow if the job env was never updated. After Path A is green on your source, Promoting to Production is how merge rewrites both ACA jobs; for a Mode 2 spike before merge, update the dev job image (and URL if you must) by hand as in the steps below.
The dev-run AZURE_CLIENT_* keys from one-time setup are what let the worker start job-fp-ingest-dev. Leave them in .env when you switch back to Mode 1.