Introduction to Orchestration

Airflow Fundamentals

Scheduling and Triggers

Sequential Pipeline Steps

Parameterized Runs and Backfills

Monitoring and Debugging

Practice

Assignment

Gotchas & Pitfalls

Week 12 Lesson Plan (Teachers)

Monitoring and Debugging

Content coming soon...

Suggested Topics

• Reading Airflow task logs: finding the root cause of failures
• The Airflow UI for debugging: task instance details, rendered templates, and XCom
• Setting up alerts: email and Slack notifications on task failure
• SLA misses: detecting when pipelines run longer than expected
• Astronomer monitoring: deployment health, DAG-level metrics, and alerting via the Astronomer UI
• Retry configuration: automatic retries with exponential backoff
• Common failure patterns: timeout errors, resource exhaustion, upstream data delays
• Building a runbook: documenting how to respond to common pipeline failures

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.