Introduction to Orchestration

Airflow Fundamentals

Scheduling and Triggers

Sequential Pipeline Steps

Parameterized Runs and Backfills

Monitoring and Debugging

Practice

Assignment

Gotchas & Pitfalls

Week 12 Lesson Plan (Teachers)

Assignment

Content coming soon...

Suggested Project Scope

• Build an orchestrated pipeline with at least two sequential steps (ingestion and transformation)
• Schedule the pipeline to run daily using a cron expression
• Parameterize the pipeline so it can be rerun for a specific date
• Implement retry logic and failure alerts on critical tasks
• Backfill the pipeline for a 7-day historical range
• Deliverables: working DAG deployed to Astronomer, screenshot of successful DAG run in the Astronomer UI, documentation of schedule and parameters
• Requirements: sequential dependencies, parameterized date handling, at least one retry configuration
• Note: retry logic is required; failure alerts (email/Slack) are a stretch goal unless a pre-configured alert endpoint is provided

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.