Week 11 - Orchestration

Introduction to Orchestration

Airflow Fundamentals

Scheduling and Triggers

Sequential Pipeline Steps

Parameterized Runs and Backfills

Testing DAGs

Monitoring and Debugging

Deploying to Shared Airflow

Practice

Gotchas & Pitfalls

Assignment: Build an Orchestrated Data Pipeline

Week 11 Lesson Plan (Teachers)

Week 11 Lesson Plan (Teachers)

This class moves students from "pipeline scripts" to "operated workflows." The goal is for every student to leave with a working Airflow DAG that runs locally, supports reruns, can be debugged confidently, and (at Target tier) deploys to the shared class Airflow.

Teaching focus for this cohort: many learners are changing careers and may still be building confidence with technical vocabulary. Keep explanations concrete, repeat key terms, and show one full example before asking students to generalize.

Goals

By the end of class, students should be able to:

• Explain orchestration vs scheduling.
• Run Airflow locally with the Astro CLI.
• Build a sequential DAG with at least three tasks.
• Parameterize a DAG with {{ ds }} / _ds_from_context() and run a small backfill.
• Debug a failed task from logs.
• Deploy a DAG to the shared class Airflow via a PR to class-airflow-reference (Target tier).

Schedule (3 hours total)

Total: 3 hours

Inclusion and accessibility strategy

• Start each segment with "why this matters at work" in one sentence.
• Use plain language first, then introduce the technical term.
• Keep instructions visible while students practice.
• Use pair roles: one driver (typing), one navigator (checking logic), then switch after 10 to 15 minutes.
• Normalize questions: remind students that asking for clarification is expected in professional teams.

Confidence checkpoints

Use quick check-ins after each major block:

1. Thumb check (30 seconds): clear / partly clear / blocked.
2. One-minute recap: one student explains the concept in plain words.
3. Reset point: if many are blocked, pause and rerun the demo slowly.

Pre-class preparation checklist

• Verify the shared Airflow VM is reachable: curl -f http://<vm-ip>:8080/api/v2/monitor/health. The VM IP changes on each bicep redeploy; get the current one from az deployment sub show --name <latest-deployment> --query properties.outputs.airflowUrl. If the VM is offline or your cohort has not provisioned one, flag that Minimum-tier students can still complete the assignment locally (see "If the shared VM is down" below).
• Confirm class-airflow-reference is pushable for the students in this cohort (grant them write access to the repo or to a cohort-specific branch).
• Confirm the student-UI passwords are distributed via the class channel. Default seed is <name>-hyf-<year> from the bicep; rotate once before class.
• Prepare one known-good demo DAG (e.g. class_demo_hello from class-airflow-reference) and one intentionally broken DAG (e.g. class_demo_hello_broken with an import typo) you can push to dags/ during the Ch8 demo.
• Prepare a short slide or whiteboard diagram for: ingest_taxi_month to dbt_run to dbt_test.

Live demo plan

Demo 1: Airflow basics (Astro UI), 25 min

1. Start local Astro: astro dev start from a scratch project.
2. Open the Airflow UI. Walk through DAG list, Graph view, Grid view, task logs, Rendered Template tab.
3. Trigger the hello_pipeline DAG (Ch2 snapshot) manually.
4. Show dependency order and successful task states.
5. Point out the Rendered Template tab on a task to preview where {{ ds }} gets substituted; this sets up the debugging demo later.

Demo 2: Debugging a real failure (part of the Debug lab at 1:50), 25 min

1. In the student workshop's DAG, inject a typo: add _typo suffix to the parquet URL.
2. Trigger the DAG. Watch ingest_taxi_month go red and dbt_run / dbt_test go upstream_failed.
3. Open the task log, scroll to the bottom, then read upward. The useful line is HTTPError: 403 Client Error: Forbidden for url:, not the generic "Task failed with exception" above it.
4. Revert the typo. Clear the failed task. Re-trigger. Everything goes green.
5. Land the takeaway: "Retries do not fix 403. You need to fix the code."

Demo 3: Deploy to the shared Airflow (Ch8 flow), 20 min

1. Starting from your local class-airflow-reference checkout, copy a class-demo DAG into dags/ (top level, above student subdirs).
2. Commit and push to a branch, open a PR, merge it.
3. Open the shared Airflow UI in the browser. Filter by the teacher tag.
4. Wait ~60 seconds: the cron on the VM runs git pull, the dag-processor re-parses, and the DAG appears.
5. Unpause, trigger. Tasks go green.
6. Land the takeaway: "This is what shared production looks like. Students will follow the same flow for their assignments."

Workshop instructions (45 min)

Students extend the Ch5 snapshot into a working taxi_pipeline on their local Astro.

Workshop targets:

• 3 sequential tasks: ingest_taxi_month to dbt_run to dbt_test.
• {{ ds }} (or _ds_from_context()) used to pick the partition in the ingest task.
• retries=2 + retry_delay=timedelta(minutes=2) on default_args.
• One manual trigger completed successfully against the shared Azure Postgres.

Coaching prompts:

• "What should happen if ingest_taxi_month fails?"
• "Where does this task get its date from?"
• "If you rerun this date, do you duplicate output?"

Support prompts for less technical backgrounds:

• "Can you explain this task as if you are training a teammate next week?"
• "Which part of this output tells you the run was successful?"
• "If this failed at 3 AM, what is your first debugging step?"

Gotchas walkthrough (10 min, just before assignment briefing)

Before Q&A, walk the class through the Gotchas & Pitfalls page at high speed. Focus on the ones students are most likely to hit on the assignment:

• Gotcha #1 (runtime vs logical date): forbids datetime.now().
• Gotcha #4 (cross-month spillover): explains why their backfill row counts may drift by 2-10 rows.
• Gotcha #6 (max_active_runs=1): do not remove it.
• Gotcha #7 (import errors are silent): run the integrity test before push.

Remind students that Ch10 ends with a 10-line "audit your own DAG" exercise they should do before submitting.

Assignment alignment

Map class activities to assignment tasks explicitly: