Parameterized Runs and Backfills
This page is optional. It covers the history behind the orchestration patterns you use this week. If you prefer to focus on the hands-on work, skip it and come back when you are curious.
Background for Chapter 1: Introduction to Orchestration and Chapter 3: Scheduling and Triggers
The first widely used job scheduler in Unix was cron, originally written by Ken Thompson for early Unix in the mid-1970s at Bell Labs. The famous five-field syntax (min hour day month weekday) dates from Paul Vixie's 1987 rewrite, and has barely changed since. Every scheduler you use today, including Airflow's schedule="0 6 * * *", speaks Vixie's dialect.
Cron was designed for one-off tasks on a single machine: rotate a log, run a backup, rebuild an index. It answers "when to start" and nothing else. If the job fails, cron does not know. If the machine reboots during the job, cron does not know. If yesterday's job is still running when today's fires, cron does not know. For decades this was fine, because most "scheduled jobs" were simple and independent.
By the mid-2000s the assumption that scheduled jobs were independent started to break. Data pipelines routinely had a dozen steps, each depending on the previous one. Cron could fire the first step on time, but had no way to express "only run step 3 if step 2 succeeded and step 1 produced output after midnight." Teams started writing wrapper scripts around cron to handle dependencies, and those wrappers gradually evolved into the first workflow engines.
Systemd's timer units (2010) and cloud-native schedulers like Kubernetes CronJobs (2017) extended cron's syntax to more modern infrastructure but kept the same fundamental model: a timer fires a command, and whatever happens after is the command's problem. That is the gap orchestrators fill.
Background for Chapter 1: Introduction to Orchestration and Chapter 4: Sequential Pipeline Steps
The idea that a computation should be expressed as a directed acyclic graph of steps is older than cron. Graph theory used DAGs to reason about scheduling and dependency order since the 1950s. The first programming tool to put this into everyday use was Make, written by Stuart Feldman at Bell Labs in 1976 as a weekend project. A Makefile declares targets, each target declares its prerequisites, and Make walks the dependency graph to decide what has to be rebuilt.
Make's core ideas, incremental execution, dependency declaration, and DAG-ordered scheduling, became the template for every orchestrator that came after it. When you write ingest >> transform >> test in an Airflow DAG, you are using a syntax that would have looked familiar to a Unix programmer in 1978. The big change is scope: Make reasons about files in a single directory; Airflow reasons about tasks across a distributed system over months of run history.
The "DAG" label for data pipelines specifically became common in the 2010s with Apache Spark's RDD lineage graphs and Hadoop's MapReduce pipelines. By the time Airbnb wrote the first line of Airflow code in 2014, "your pipeline is a DAG" was already the obvious framing.
Background for Chapter 1: Introduction to Orchestration
Between 2008 and 2014, several companies hit the same problem at the same time: Hadoop jobs were getting too complicated for cron. Each company built its own workflow engine:
requires() and output() methods. Luigi ran hundreds of production pipelines at Spotify before Airflow existed. It is still maintained, and its influence on Airflow's Python-first design is direct.All four followed the same pattern: define your pipeline in code or config, let the engine walk the DAG, capture failures, expose a UI. The differences were in language, configuration format, and integration ecosystem. The one that won was Airflow, largely because it shipped a richer UI and more operators out of the box.
Background for Chapter 2: Airflow Fundamentals
Maxime Beauchemin started Airflow at Airbnb in October 2014, open-sourced it in June 2015, and donated it to the Apache Software Foundation in March 2016. It became a top-level Apache project in January 2019.
Two decisions shaped Airflow's trajectory:
By 2019, Airflow was the default choice for data orchestration at most companies that had outgrown cron but were not on a closed-source cloud service. It ran at Airbnb, Netflix, Lyft, Adobe, Slack, and Twitter, often scheduling tens of thousands of DAGs.
Background for Chapter 2: Airflow Fundamentals and Chapter 6: Testing DAGs
Airflow has gone through three major versions. Each one is a direct response to a pain point the previous version caused at scale.
| Version | Released | Key change | Why |
|---|---|---|---|
| 1.0 | 2015 | First public release | Airbnb needed to share it |
| 2.0 | 2020 | Providers split out, TaskFlow API, scheduler HA | Operator bloat, single-scheduler bottleneck |
| 3.0 | 2025 | airflow.sdk, mandatory dag-processor, REST API v2, FAB provider |
DAG authoring simplification, scheduler isolation |
Airflow 2.0 (December 2020) split the ~700 built-in operators into separately versioned provider packages (apache-airflow-providers-postgres, apache-airflow-providers-amazon, and so on). This let providers release on their own cadence instead of waiting for an Airflow core release. It also introduced the TaskFlow API (@task decorators) which is what you write in this week. The older operator-as-class style (PythonOperator(python_callable=...)) still works but reads awkwardly by modern standards.
Airflow 3.0 (2025) was a bigger reset. It introduced the airflow.sdk import path you use in every Week 12 DAG, renamed webserver to api-server, made the dag-processor service mandatory (opt-in since 2.3), replaced the session-based UI auth with JWT, and moved authentication into the FAB provider. If you read older Airflow tutorials online, most of the imports are wrong for Airflow 3. The historical reason the curriculum pins to Airflow 3 is that the upgrade happens once every few years; new courses should not teach the version about to be deprecated.
The "Airflow 3 compatibility" work scattered through this week's chapters (the airflow.sdk imports, the _ds_from_context() helper for manual triggers, the dag-processor callouts in Testing DAGs) exists because the move from 2.x to 3.x broke enough surface area that it shows up in student code. Five years from now, none of that will matter.
Background for Chapter 4: Sequential Pipeline Steps and Chapter 10: Gotchas & Pitfalls
The word idempotent is borrowed from mathematics: a function f is idempotent if f(f(x)) = f(x). Applied to pipelines, it means "rerunning a step produces the same correct result." That property is not interesting until you try to rerun a failed pipeline, at which point it becomes the single most important design constraint.
The data community's modern focus on idempotency traces to Nathan Marz's Lambda architecture (described in his 2015 book Big Data). Lambda split pipelines into a batch layer that reprocesses everything from source-of-truth raw data and a speed layer that handles real-time updates. The batch layer only works if every transformation is idempotent: you have to be able to drop the output tables and rebuild them from scratch without getting different answers.
The Lambda architecture itself fell out of fashion by 2018, replaced by the simpler Kappa architecture (streaming only) and then by the lakehouse model (one lake, batch and streaming share the same storage). But Nathan Marz's insistence on idempotent transformations stuck. Every modern orchestrator, including Airflow, assumes your tasks are idempotent. Retries and backfills only work if they are. The DELETE-then-INSERT pattern you use in taxi_pipeline.py is the most common way to enforce it.
Background for Chapter 1: Introduction to Orchestration
Airflow is still the most widely deployed orchestrator, but it no longer has the field to itself. Three newer tools cover the same problem space with different trade-offs:
Dagster reframes the primary unit from "task" to "asset" (a specific table or file the pipeline produces). Asset-aware orchestration lets Dagster know what a DAG is supposed to materialize, not just what order to run steps in.
The single concrete reason Dagster caught on with modern data teams is its dbt integration. The dagster-dbt library turns each dbt model into a Dagster asset with its own lineage node. From the official docs: "Dagster assets understand dbt at the level of individual dbt models." In practice that means you can see a single graph in the Dagster UI that spans your Python ingestion code, your dbt transformations, and your downstream ML or reverse-ETL steps, with each dbt model surfaced as a distinct asset rather than a single opaque dbt run task. Dagster's own blog on orchestrating dbt is the canonical explainer, and the integration landing page lists lineage, model-level scheduling, partition-aware runs, and per-model failure isolation as the headline benefits.
In Airflow, dbt run is a single BashOperator call by default (that is what you do this week). You get one success/failure line per dbt invocation. Dagster gets you per-model success/failure and per-model lineage without any extra code. For data teams where dbt is already the spine of the warehouse, that granularity is the whole reason to switch.
Prefect positioned itself as "Airflow that does not need a server." It keeps the "workflows are Python functions" idea from Airflow's TaskFlow API but removes two constraints Airflow imposes.
First, boilerplate. Prefect docs promise you can "write workflows in native Python, no DSLs, YAML, or special syntax." A Prefect flow is just a Python function with @flow. There is no dag_id, no start_date, no scheduler config unless you want one.
Second, static DAGs. Airflow parses your DAG file once to build a fixed graph before any task runs. You cannot truly branch on a value you only know at runtime. Prefect 3 explicitly dropped that: "we removed a key constraint that workflows be written explicitly as DAGs, fully embracing native Python control flow: if/else conditionals, while loops" (Prefect 3 flows docs). If your pipeline needs to loop until an API returns "done," or branch based on a row count computed earlier in the run, Prefect handles that with ordinary Python. In Airflow you reach for @task.branch, dynamic task mapping, or TriggerDagRunOperator and accept some awkwardness.
Third, hybrid execution. Prefect Cloud hosts the scheduler, API, and UI; your workflow code runs on your infrastructure. That gives small teams the "managed orchestrator" experience without exposing their data or network to Prefect, and without needing to operate a scheduler themselves. For teams of one to five engineers, avoiding Airflow's cluster-operations overhead is often the deciding factor.
Temporal is a general-purpose workflow engine (not data-specific) that powers microservice workflows at Uber, Netflix, Snap. Not a direct competitor to Airflow but the underlying primitives (durable execution, retries, idempotency) overlap heavily.
Managed orchestrators also became a real option: Azure Data Factory, AWS Step Functions, Google Cloud Composer (hosted Airflow), Databricks Workflows. For teams on a single cloud, the managed options remove the "who runs the scheduler" question entirely.
The course teaches Airflow because it is still the most common choice in Dutch data teams, and the concepts (DAGs, idempotency, logical dates, backfills) transfer directly to every tool above. Switching from Airflow to Dagster, Prefect, or ADF is a syntax change, not a conceptual one.
Background for Chapter 1: Introduction to Orchestration
Looking at the whole arc: each generation of orchestrator solved a problem the previous generation created.
dag-processor mandatory.Five years from now there will be another round. The durable skill is not the specific tool but the mental model: your pipeline is a DAG, your tasks are idempotent, retries and backfills are first-class, and failure visibility is non-negotiable. Whatever the next orchestrator looks like, it will respect those four ideas because they are what makes data pipelines operable at all.
Each row is one generation of the story above: what the technology solved, and the gap it left for the next generation to close.
| --- | --- | --- | --- |
airflow.sdk split and the mandatory dag-processor.