Introduction to Orchestration
Airflow Fundamentals
Scheduling and Triggers
Sequential Pipeline Steps
Parameterized Runs and Backfills
Monitoring and Debugging
Practice
Assignment
Gotchas & Pitfalls
Week 12 Lesson Plan (Teachers)
Parameterized Runs and Backfills
Content coming soon...
Suggested Topics
- Why parameterization matters: reprocessing historical data without changing code
- Airflow params and macros: passing execution_date, ds, and custom parameters to tasks
- Templating with Jinja: using {{ ds }} and {{ params.my_param }} in operators
- Backfills: rerunning a DAG for a historical date range using the CLI
- Clearing tasks: rerunning specific tasks or date ranges from the UI
- Idempotent pipelines: designing tasks that produce the same result when re-executed
- Date partitioning: processing data one day at a time for reliable incremental loads
- Practical example: backfilling a week of data after fixing a bug
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*
Found a mistake or have a suggestion? Let us know in the feedback form.