Introduction to Orchestration

Airflow Fundamentals

Scheduling and Triggers

Sequential Pipeline Steps

Parameterized Runs and Backfills

Monitoring and Debugging

Practice

Assignment

Gotchas & Pitfalls

Week 12 Lesson Plan (Teachers)

Parameterized Runs and Backfills

Content coming soon...

Suggested Topics

• Why parameterization matters: reprocessing historical data without changing code
• Airflow params and macros: passing execution_date, ds, and custom parameters to tasks
• Templating with Jinja: using {{ ds }} and {{ params.my_param }} in operators
• Backfills: rerunning a DAG for a historical date range using the CLI
• Clearing tasks: rerunning specific tasks or date ranges from the UI
• Idempotent pipelines: designing tasks that produce the same result when re-executed
• Date partitioning: processing data one day at a time for reliable incremental loads
• Practical example: backfilling a week of data after fixing a bug

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.