Introduction to Orchestration

Airflow Fundamentals

Scheduling and Triggers

Sequential Pipeline Steps

Parameterized Runs and Backfills

Monitoring and Debugging

Practice

Assignment

Gotchas & Pitfalls

Week 12 Lesson Plan (Teachers)

Gotchas & Pitfalls

Content coming soon...

Suggested Topics

• Execution date confusion: Airflow's execution_date refers to the start of the interval, not when the task actually runs
• Catchup surprises: enabling a new DAG with catchup=True can trigger hundreds of backfill runs
• Non-idempotent tasks: tasks that append instead of upsert produce duplicates on rerun
• Hardcoded dates: using datetime.now() instead of Airflow's templated dates breaks backfills
• Connection management: forgetting to configure Airflow connections leads to cryptic import errors
• Resource exhaustion: running too many tasks in parallel can overwhelm your database or API
• Astronomer deployment lag: after astro deploy, DAGs may take a minute to appear in the UI
• Astronomer free tier limits: be aware of DAG count and task concurrency limits on trial accounts

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.