Parameterized Runs and Backfills
In Weeks 3 through 10, you built important pipeline pieces: ingestion, transformations, tests, and cloud deployment patterns. In production, those pieces must run in the right order, at the right time, and with clear failure visibility. That coordination layer is called orchestration.
By the end of this chapter, you should be able to:
Orchestration coordinates pipeline tasks, dependencies, retries, and run history across your full workflow.
Think about a daily flow:
If one step fails, the next step should not run. If a transient API failure occurs, a retry should happen automatically. If a run misses its window, someone should know. Orchestration handles all of this.
<aside> 💡 Scheduling answers "when to start." Orchestration answers "what to run, in what order, with what recovery behavior."
</aside>
Manual execution works for experiments, but it breaks in production:
In data work, a "small" miss can become a business incident: stale dashboards, wrong KPIs, or incomplete model outputs.
<aside> ⌨️ Hands on: Think of a pipeline you built in an earlier week (e.g. Week 10's taxi dbt pipeline). Write down the sequence of steps that run, and trace what happens if the ingestion step fails: does downstream dbt run anyway? How long does it take you to find out it failed?
</aside>
Picture a plain cron job. At 06:00 it fires one command and walks away. It never checks whether the command finished, whether it crashed halfway, or whether yesterday's run left your tables in a strange state. Cron answers a single question: when should this start?
An orchestrator stays in the room. It runs your pipeline as a graph of steps and watches every one. When a step depends on the one before it, the orchestrator waits for that step to succeed first: that order relation is a dependency. When a step fails because an API blipped or a connection dropped, it tries again on its own, an automatic retry. It records every run and keeps the logs for each step, then shows all of it in a web UI so you can see at a glance what ran, what broke, and why. And when you need to reprocess an old date, you ask it to rerun that day, a backfill, instead of editing dates by hand.
flowchart LR
subgraph Cron["06:00 · Cron"]
direction LR
C1["run script.py"]
end
subgraph Orchestrator["06:00 · Orchestrator"]
direction LR
O1["ingest"] --> O2["transform"]
O2 --> O3["test"]
O3 --> O4["notify"]
O1 -. retry on<br/>transient failure .-> O1
O3 -. skip on<br/>upstream failure .-> O4
end
The cron side is a single black box: one command, no visibility, no recovery. The orchestrator side is a graph of tasks with explicit edges for success, failure, and retry.
You have already met most of these in the story above. Here they are in one place, so the vocabulary is set before the next chapter:
Those six words describe one shape: tasks joined by dependency arrows into a single execution order. This short animated visual walks through that shape for a simple three-task chain.
<aside> 🤓 Curious Geek: Why DAGs are "acyclic"
In graph theory, "acyclic" means "no loops." Airflow DAGs cannot contain cycles because cyclic dependencies have no valid execution order. This design forces pipelines to be explicit and executable.
</aside>
This is a preview list. Later chapters unpack idempotency and backfills in detail; treat the definitions above as breadcrumbs, not the full story.
<aside> ⚠️ If tasks are not idempotent, retries and backfills can create duplicate or corrupt outputs.
</aside>
By Week 12, your stack looks like this:
flowchart TB
subgraph pipeline["Pipeline"]
direction LR
src[("Source<br/>API / files")] --> ing["Ingestion"]
ing --> sto[("Storage<br/>Postgres / Blob")]
sto --> dbt["dbt models<br/>+ dbt tests"]
dbt --> bi["BI dashboard"]
end
orch["<b>Orchestration layer</b><br/>schedule · retries · logs · alerts"]
orch -.-> ing
orch -.-> sto
orch -.-> dbt
orch -.-> bi
classDef layer fill:#fff4e6,stroke:#e08e45,stroke-width:2px,color:#333;
class orch layer;
Without orchestration, each step might be correct in isolation but unreliable as a system.
<aside> 💡 Using AI to help: Paste your pipeline's steps and logic (⚠️ Ensure no PII or sensitive company data is included!) into an LLM and ask: "Which step in this pipeline is most likely to fail, and what failure gates or retries should I add to keep it reliable?"
</aside>
Picture the pipeline you will build this week. Every morning at 06:00 UTC, a fresh slice of NYC taxi trips (the same dataset you have carried since Week 9) needs to land in your warehouse, cleaned and tested, before anyone opens the dashboard.
The orchestrator wakes up on schedule and, in one ingest task, pulls that run's trip file from the public taxi dataset and loads those raw trips into your Postgres tables. Only once that load succeeds does it run dbt run to rebuild your staging and mart models, turning raw trips into fct_trips and the borough summaries the dashboard reads. Then dbt test checks the result: no negative fares, no duplicate trip ids, no trips pointing at a zone that does not exist. If any task fails, the run stops right there and the task turns red in the UI, instead of quietly leaving a stale dashboard for someone to find at noon.
That is three tasks in a strict order with a failure gate between each one, exactly the job an orchestrator exists to do. This chapter gives the "why." The next chapters show the "how."
You work in two environments:
| Environment | Purpose | Tooling |
|---|---|---|
| Local machine | Build and test DAG code | Astro CLI (astro dev start) |
| Shared class VM | Demo and teacher review | Airflow + Docker Compose |
Always develop and iterate on your DAG code locally first. Use the shared class VM only for demos and teacher review, not as your primary development environment.
The same problem shows up at much larger scale in production codebases.
<aside> 💡 In the wild: Airflow itself was open-sourced by Airbnb in 2015 to solve exactly this problem at scale. Browse the standard-provider example DAGs to see how the project's own maintainers wire up dependencies, retries, and trigger rules. The example DAGs are the canonical reference for "what good looks like": they ship with every install.
</aside>
<aside> 🚀 Try it in the widget: Interactive Quiz: Introduction to Orchestration (covering orchestration vs scheduling, DAGs, idempotency, and when manual scripts are still fine).
</aside>
https://lasse.be/simple-hyf-teach-widget/mcq.html?bank=week_12_ch1_intro_orchestration_quiz&embed=1
If the line between scheduling and orchestration still feels fuzzy, this short video walks through it from a different angle.
<aside> 🎬 Struggling with this concept? Watch this beginner-friendly video:
</aside>
https://www.youtube.com/watch?v=eum5CxOUnEU
The next chapter installs Airflow locally and has you run a DAG, so it assumes the vocabulary is in place. No state to check yet: just confirm the mental model.
Next up: Airflow Fundamentals, where you install Airflow locally with the Astro CLI and run your first DAG through the web UI.