Parameterized Runs and Backfills
A pipeline is not production-ready until you can handle failures. On a green run you ignore the Airflow UI; on a red run it is your only lever. This chapter teaches you how to pull that lever: find the failing task, read its logs, decide whether retries will help, and write down what you learned so the next person (future you, at 03:00) does not have to rediscover it.
By the end of this chapter, you should be able to:
When a DAG fails, first inspect:
{{ ds }}, {{ conn.azure_pg.host }}, and the other Jinja placeholders you wrote in Parameterized Runs and Backfills. When a task fails with "host is None" or "file not found for 2024-13", the rendered tab usually tells you why in one glance.This order usually shows whether the issue is in data, configuration, dependency, or infrastructure layers.
Task logs often show the failing line. Focus on:
try 1 of N)Do not read only the last line. The main clue is often earlier.
Retries help with transient failures:
Example (see Airflow's default_args reference for the full list of retry-related keys):
default_args = {
"retries": 2,
"retry_delay": timedelta(minutes=5),
}
Retries do not fix deterministic code bugs (bad SQL, missing column, wrong path). Those need code changes.
<aside> ⚠️ Too many retries can hide real failures and increase system load.
</aside>
When tasks do not start at all (the UI shows them stuck in queued, or the DAG is missing), the problem is below the DAG code: a scheduler or dag-processor container is down or crashing. For the Astro local stack introduced in Airflow Fundamentals:
astro dev ps # container health
astro dev logs scheduler # recent scheduler activity
astro dev logs dag-processor # parsing errors
<aside>
💭 Teacher-only (needs SSH to the VM). On the shared class VM the Astro wrapper is not available, so the equivalent commands are raw Docker Compose. Students have no SSH access to the VM, so you debug shared runs through the UI (see Deploying to Shared Airflow); these are shown so you know the local astro dev commands map onto the same containers.
</aside>
docker compose ps # replaces `astro dev ps`
docker compose logs -f scheduler # replaces `astro dev logs scheduler`
Abstract failure taxonomies are easy to nod along to and forget. Here are the two failure modes to know on the Week 12 taxi_pipeline DAG from Sequential Pipelines, with the log pattern each produces and the fix. You can reproduce the first on demand (the hands-on below does exactly that); the second is what a data-quality failure looks like when a bad month slips through.
What it looks like in the UI. ingest_taxi_month goes red in the grid view; dbt_run and dbt_test go grey (upstream_failed). The grid's per-task rail lights up the downstream cascade, and the Task Instances table lists each task's state:
Airflow grid view of a failed taxi_pipeline run: ingest_taxi_month red Failed, dbt_run and dbt_test both Upstream Failed; the Task Instances table on the right lists each task and its state
Click the red ingest cell and open the Logs tab. The real log from a 403 (here, a month TLC has not published yet) looks like this (captured against the live TLC endpoint):
Airflow task log panel: ERROR line reads HTTPError 403 Client Error Forbidden for url https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2026-07.parquet with the full traceback below
Diagnose by reading upward from the requests.exceptions:
_typo suffix or a 2026-07 that TLC has not yet published, CloudFront refuses to sign the asset and you get HTTPError: 403 Client Error: Forbidden./trip_data/ instead of /trip-data/).HTTPError: line, not the generic "Task failed with exception" on the line above.Fix. For 403: audit the ds[:7] slice; confirm catchup is not creating runs for future dates; narrow the end_date of the backfill to last month if the current month is not yet published. For 404: copy the URL from the log into curl -I and verify which segment of the path is wrong.
Why retry alone will not rescue this. retries=2 re-runs the task, which re-issues the same GET to the same URL. The HTTP error is deterministic against the URL, so retries only waste time. Retries help with transient network failures (DNS blip, connection reset), not 4xx responses.
<aside>
💡 Notice the log line File ".../requests/models.py", line 1167 in raise_for_status. This tells you the reference DAG calls response.raise_for_status() on every TLC download, which converts a silent 403/404 HTML body into a real Python exception. Without it, the HTML error page would be fed straight to pd.read_parquet inside the same task and you would see a cryptic ArrowInvalid: Parquet magic bytes not found instead: harder to diagnose because it points at pandas rather than at the real culprit (the HTTP response).
</aside>
Against the clean January-2024 data the suite is green: dbt_test ends Done. PASS=8 WARN=3 ERROR=0 (the three warnings are the Week-10 duplicates, null payment_types, and orphaned pickup zones from Sequential Pipelines). This failure mode is what you would see if a month slipped in rows that break an error-severity test: the accepted_values test on payment_type is configured to fail (not warn), so a code outside [1,2,3,4,5,6] turns the whole run red.
What it looks like in the UI. ingest_taxi_month and dbt_run both green. dbt_test red. Downstream consumers (any dashboard DAG that depends on fct_trips) are blocked, which is exactly what the failure gate exists for.
Log pattern (the three usual warnings, plus the one test that flipped to a hard failure):
[2026-04-22 06:03:41] INFO - Running: dbt test --project-dir /usr/local/airflow/include/dbt_project
...
3 of 11 WARN 4 dbt_utils_unique_combination_of_columns_stg_trips_... [WARN 4 in 0.68s]
5 of 11 WARN 3415 not_null_stg_trips_payment_type ... [WARN 3415 in 0.58s]
8 of 11 FAIL 47 accepted_values_stg_trips_payment_type__1__2__3__4__5__6 [FAIL 47 in 0.52s]
9 of 11 WARN 5 relationships_stg_trips_pickup_location_id ... [WARN 5 in 0.40s]
Done. PASS=7 WARN=3 ERROR=1 SKIP=0 NO-OP=0 TOTAL=11
[2026-04-22 06:03:42] ERROR - Task failed: dbt test exited with code 1
Diagnose from the dbt-test summary:
WARN lines (4 duplicates, 3415 null payment_types, 5 orphaned pickup zones) are the expected Week-10 data-quality findings; warnings do not fail the task, so they are not the cause.FAIL 47 on accepted_values_stg_trips_payment_type is the cause: 47 rows have a payment_type outside the documented [1,2,3,4,5,6]. Either the TLC added a new code this month or the ingest mangled the type. Open the compiled query from target/compiled/... and run it against stg_trips in psql to see the offending values.Fix. If the TLC truly added a new code, update accepted_values in _stg_trips.yml to include it. If the ingest is the culprit (e.g. integer parsed as string), fix the type coercion in the load task and rerun the affected month.
Why this is the ideal orchestration win. The four-task split you built in Sequential Pipelines means the broken dbt_test does not contaminate downstream dashboards: they are blocked by the failure gate until a human resolves the data-quality issue. A single-script version of this pipeline would have quietly shipped bad data.
Three topics matter in production but are out of scope for a Week 12 project and live in the optional Going Further page:
execution_timeout tuning (a per-task deadline Airflow enforces by killing the task), resource pressure, and upstream-data delay. Worth skimming when you plan a real on-call rotation.Import errors are the one category you do already have a guard for: the integrity test from Testing DAGs catches them before they reach the scheduler.
A short runbook should include:
This reduces panic and helps teams respond in a consistent way.
<aside> 💭 Treat runbooks as living documents. Update them after important failures.
</aside>
The best runbooks are short enough to use under pressure.
Work through the full incident-handling loop: break → observe → diagnose → fix → verify. Each step maps to a real action an on-call engineer takes.
Step 1: Break the download URL. In dags/taxi_pipeline.py, find the parquet_url_for helper and add a _typo suffix to the filename:
# noqa: verify (illustrative before/after fragment; return shown outside its function)
# before
return f"{TLC_BASE}/green_tripdata_{year_month}.parquet"
# after (break it)
return f"{TLC_BASE}/green_tripdata_{year_month}_typo.parquet"
Step 2: Trigger a run. Either click Trigger in the UI or from your project root run:
astro dev run dags trigger taxi_pipeline
Step 3: Observe the failure in the UI. Open the DAG page. You should see one red task (ingest_taxi_month) and two orange upstream_failed tasks (dbt_run, dbt_test), matching the first screenshot above. The Failed Task and Failed Run counters on the overview both read 1.
Step 4: Read the log upward. Click ingest_taxi_month, open Logs. Scroll to the bottom and then read upward from the last frame: the useful line is the HTTPError: 403 Client Error: Forbidden for url: ... line, not the generic "Task failed with exception" above it. Note which URL got requested.
Step 5: Check the Rendered Template tab (for any failing task with Jinja templating). For dbt_run / dbt_test, the Rendered Templates tab shows what Airflow actually substituted for {{ conn.azure_pg.host }} and the other DBT_ENV placeholders. This is where you verify "is Airflow seeing the connection I think it is?" when a DB-connection task fails:
Airflow Rendered Templates tab for dbt_run: the bash_command shows dbt invoked through uvx (uvx --python 3.11 --from dbt-core 1.10 --with dbt-postgres 1.10 dbt run --project-dir and so on), and the env dict shows PG_DBNAME substituted to team1, PG_HOST to hyf-data-pg with its middle segment masked, PG_SCHEMA to airflow_reviewer, PG_USER to reviewer, and PG_PASSWORD redacted
The password is redacted (and Airflow masks the host's middle segment as ***); the database, schema, and user appear substituted verbatim. Your own run shows airflow_<your-role> in place of airflow_dev_demo. The tab is empty for tasks that never ran (upstream_failed state) and for pure-Python @task tasks that read context directly instead of relying on Jinja, so do not be alarmed when a failed task's rendered tab looks blank.
Step 6: Write a runbook entry. Create RUNBOOK.md next to your DAG and add:
## ingest_taxi_month fails with HTTPError 4xx
**Symptom:** ingest_taxi_month goes red; dbt_run / dbt_test go upstream_failed.
**Check:** Task log, read upward from the last traceback frame. The first useful line is `HTTPError: <code> ... for url: <url>`.
**Fix:** If 403, the TLC path is wrong or the month is not yet published. If 404, a path segment is mistyped. Copy the URL into `curl -I` to confirm.
Step 7: Revert the typo and trigger once more. Confirm the DAG goes fully green (all three tasks success). Idempotency check: re-trigger the same logical date a second time without changing anything. You should get the same green DAG, because ingest_taxi_month starts with DELETE FROM ... WHERE year_month = %s and re-inserts the same rows. No drift, no duplicates.
Completing this loop once gives you the muscle memory the runbook is trying to encode.
Operational maturity grows through repeated, structured incident handling.
<aside> 🤓 Curious Geek: "Toil" in SRE
</aside>
You can speed up triage notes with AI, but keep human judgment for final actions.
<aside> 💡 Using AI to help: Paste a sanitized log excerpt (⚠️ Ensure no PII or sensitive company data is included!) and ask for likely root-cause categories before you decide the final fix.
</aside>
You reproduced a failure end-to-end and drafted a runbook with the hands-on above. One more retry detail is worth trying by hand.
<aside>
⌨️ Hands on: The reference taxi_pipeline sets default_args={"retries": 2} at the DAG level, so every task inherits two retries. Override it per task: add retries=4, retry_delay=timedelta(minutes=1) to ingest_taxi_month only, leaving dbt_test on the DAG-level default. In one sentence, say why generous retries belong on the network-bound ingest task but are pointless on the test task.
</aside>
The week's practice chapter puts this to work on a fresh failure you have not seen.
<aside> 📝 Practice: The week's Practice chapter has an exercise on debugging a deliberately broken DAG run. It reinforces the log-reading + runbook pattern against a fresh failure.
</aside>
Production teams go further than the Grid view + log-reading you practised here.