Parameterized Runs and Backfills
In Airflow Fundamentals your hello_pipeline ran on schedule="@daily" and you triggered it manually. That glossed over three things Airflow decides for you: when the next run fires, which historical runs get created when you unpause, and which date each run represents. This chapter turns those three into deliberate choices you make with cron expressions, catchup, and logical-date semantics.
You will edit hello_pipeline throughout the chapter rather than build something new. Every hands-on swaps one parameter on the DAG you already have and watches what changes.
By the end of this chapter, you should be able to:
schedule=.{{ ds }} (logical date) is different from datetime.now() and when that difference matters.catchup=True safely, and predict how many historical runs it will create before you unpause the DAG.In the previous chapter, you used a blank week12-airflow sandbox directory to learn how Astro CLI initializes a project. For the rest of this week, you will work inside the official class reference repository (nyc-taxi-airflow-reference). This repository contains the reference dbt project, companion datasets, SQL schemas, and python dependency lockfiles pre-configured for you.
Follow these steps to set up and switch to the reference repository before proceeding:
Step 1: Stop your sandbox stack. In your week12-airflow directory, run:
astro dev stop
Step 2: Clone the reference repository. Navigate out of your sandbox folder and clone the repository:
git clone <https://github.com/lassebenni/nyc-taxi-airflow-reference>
cd nyc-taxi-airflow-reference
Step 3: Set up python dependencies (optional, IDE only). If you have uv installed, run uv sync so your editor gets Airflow autocomplete from the project's lockfile. Skipping it is fine: this step does not install packages into the Airflow containers. Container packages still come from requirements.txt via astro dev restart. On Windows without uv, skip this step and keep writing DAGs; Astro and Docker are what run the stack.
Step 4: Copy your DAG. Copy the hello_pipeline.py file you wrote from your sandbox dags/ folder into the dags/ folder of the newly cloned nyc-taxi-airflow-reference directory.
Step 5: Start the reference stack. Start the Airflow containers inside the nyc-taxi-airflow-reference folder:
astro dev start
From now on, run all CLI commands and write all DAGs inside this nyc-taxi-airflow-reference directory.
An Airflow DAG can run with:
"0 6 * * *"): full control over minute, hour, day-of-month, month, day-of-week."@daily", "@hourly", "@weekly", "@monthly"): readable shortcuts for the common patterns.schedule=None): the DAG runs only on manual trigger or via another DAG.The hello_pipeline from Airflow Fundamentals uses the @daily preset. Let us switch it to a real cron expression and watch the UI update:
@dag(
schedule="0 6 * * 1-5", # was "@daily"
start_date=datetime(2025, 1, 1),
catchup=False,
tags=["week12", "intro"],
)
def hello_pipeline():
...
This runs every weekday at 06:00 UTC. No weekend runs.
Work through the schedule change in your nyc-taxi-airflow-reference Astro stack:
Step 1: Edit the cron expression. In dags/hello_pipeline.py (copied from Airflow Fundamentals), change schedule="@daily" to schedule="0 6 * * 1-5" as in the snippet above. Save the file.
Step 2: Reserialize. From the project root, force an immediate re-parse:
astro dev run dags reserialize
Step 3: Open the DAG list. With astro dev start running in nyc-taxi-airflow-reference, open the Airflow UI at http://nyc-taxi-airflow-reference.localhost:6563/dags (if Astro printed a different port in your terminal, use that URL instead). Find the hello_pipeline row and read the Next Run column.
Airflow Dags page: locate hello_pipeline and read the Next Run column after reserialize
After the change, Next Run should land on the next upcoming weekday at 06:00 UTC, not tomorrow at midnight. If the column still shows a daily midnight time, wait up to 60 seconds and refresh, or re-run the reserialize command.
Step 4: Open the Runs tab. Click the hello*pipeline name, then open the Runs tab, or go directly to http://nyc-taxi-airflow-reference.localhost:6563/dags/hello*pipeline/runs. This page lists every execution of the DAG: manual runs you triggered earlier, plus any scheduled runs the scheduler has queued.
Airflow Runs tab for hello_pipeline showing run history and run types
Step 5: Confirm the schedule stuck. On the DAG list row, Next Run should still show the next weekday at 06:00 UTC. On Runs, new scheduled entries (when they appear) should respect the weekday-only cadence: no Saturday or Sunday dates for 0 6 * * 1-5. Manual runs from Chapter 2 remain in the table with run_type=manual; that is expected.
| Pattern | Meaning |
|---|---|
0 * * * * |
every hour at minute 0 |
0 6 * * * |
every day at 06:00 |
0 6 * * 1 |
every Monday at 06:00 |
0 6 * * 1-5 |
weekdays at 06:00 |
0 0 1 * * |
first day of every month at midnight |
*/15 * * * * |
every 15 minutes |
The five fields are minute hour day-of-month month day-of-week. For anything non-trivial, paste your expression into crontab.guru: it shows the next 5 run times in plain English, which catches off-by-one errors before you ship.
<aside>
💡 Start with a daily schedule while learning. Run more often only when your pipeline is stable. A */5 * * * * DAG that fails once an hour generates 12 failure notifications before breakfast.
</aside>
Airflow injects a logical date into every task: the Jinja template {{ ds }}, or the ds argument in a TaskFlow task. It represents the data interval the run is processing, not the wall-clock time the run started.
Here is the idea in plain terms. Think of a daily newspaper. Each edition is dated for the day it covers, no matter when it actually reaches you. If the delivery truck breaks down and the paper arrives a day late, it is still that day's edition, not the next day's. Airflow treats each run the same way: the slice of data a run owns is its partition (one day, one month, one hour), and {{ ds }} names that partition, not the wall-clock moment the run executes.
You will see this directly in the catchup exercise below, where you generate a batch of runs in one burst: one per weekday, each carrying a different {{ ds }}. The screenshot there makes the split obvious. All 23 runs share a Start Date within the same minute (they were all created at once), yet each owns a different logical date. The logical date is what a run is for; the Start Date is when it ran.
This matters any time tasks read or write date-partitioned data:
# noqa: verify (illustrative snippet; read_partition / write_to are stand-ins)
@task()
def load_yesterday(ds: str):
# ds is the logical date, e.g. "2024-01-15"
df = read_partition(ds) # correct: partition identity
write_to(f"output/{ds}.csv") # correct: deterministic path
bad = datetime.now() # WRONG: breaks backfills and late runs
datetime.now() changes every time the task runs, even for the same scheduled date. Backfilling yesterday's partition with datetime.now() writes today's date into the file path: silent corruption.
<aside>
⚠️ Using datetime.now() as a partition key is the single most common idempotency bug in Airflow code. Reach for ds, data_interval_start, or data_interval_end instead: key every path, filter, and download URL off the logical date the run is for, not the moment the worker happens to execute, as in the load_yesterday(ds) example above.
</aside>
catchup=True tells Airflow: "when this DAG is unpaused, create runs for every scheduled interval between start_date and now."
That sentence sounds innocent. It is not. Concrete example: a @daily DAG with start_date=datetime(2024, 1, 1) that you unpause on 2026-04-22 produces 843 historical runs, all queued at once. If each run takes 30 seconds, the full catchup takes about 7 hours.
@dag(
schedule="@daily",
start_date=datetime(2024, 1, 1),
catchup=True, # creates 843 historical runs on unpause
)
def daily_pipeline():
...
catchup=False is the safe default: Airflow only runs from now forward, ignoring anything that would have fired between start_date and the present.
Airflow Runs tab for hello_pipeline showing 23 historical runs generated by catchup on a weekday-only schedule
The screenshot above is the result you will produce in the walkthrough below: hello_pipeline with a past start_date and catchup=True lands 23 historical successful runs in one burst. Notice the run dates skip the weekends: Apr 25-26 sits in the gap between the 04-24 and 04-27 runs, and Apr 18-19 between the 04-17 and 04-20 runs. The cron 0 6 * * 1-5 only fires on weekdays, and catchup respects the schedule. Because hello_pipeline does no real work, every run goes green in seconds.
Now reproduce it on your own hello_pipeline, step by step.
Step 1: Pause and clear first. Open http://nyc-taxi-airflow-reference.localhost:6563/dags/hello_pipeline. Flip the pause toggle next to the DAG name off (paused), then open the Runs tab, select any existing runs, and delete them. This matters: if the DAG still has a successful run at "now," Airflow's catchup logic treats that as the latest run and skips the historical interval entirely. Clearing gives catchup a clean slate.
Step 2: Set a bounded catchup window. In dags/hello_pipeline.py, change the @dag decorator to:
@dag(
schedule="0 6 * * 1-5",
start_date=datetime(2026, 4, 1),
end_date=datetime(2026, 5, 2), # bounds the window: run count stays fixed
catchup=True,
tags=["week12", "intro"],
)
The end_date bounds the window so the run count stays fixed at 23 no matter which day you do the exercise. Without it, catchup keeps generating new runs every day the exercise sits open, and the number drifts.
Step 3: Reserialize. From the project root, force an immediate re-parse:
astro dev run dags reserialize
Step 4: Unpause and watch. Back on the hello_pipeline page, flip the pause toggle on. Within a minute the Runs tab fills with one run per weekday between April 1 and May 1: 23 runs, matching the screenshot above (every weekday in April plus May 1; weekends skipped).
Step 5: Reset for later chapters. Set catchup=False, remove the end_date line, reserialize, and clear the 23 historical runs from the Runs tab. Later hands-on assume hello_pipeline starts from a clean run history, so leaving these runs around will throw off the run counts you are asked to verify.
Rule of thumb: keep catchup=False while learning. Flip it to True only when:
The Calendar view renders each scheduled run as a dot on a month grid. For a @daily DAG you see 30 dots in a row; for 0 6 * * 1-5 you see dots on weekdays only, gaps on weekends. Open it at http://nyc-taxi-airflow-reference.localhost:6563/dags/hello_pipeline/calendar (if Astro printed a different port in your terminal, substitute that host).
Airflow Calendar view showing scheduled weekday runs across April 2026, with dots on Mon-Fri only