Introduction to Orchestration

Airflow Fundamentals

Scheduling and Triggers

Sequential Pipeline Steps

Parameterized Runs and Backfills

Monitoring and Debugging

Practice

Assignment

Gotchas & Pitfalls

Week 12 Lesson Plan (Teachers)

Sequential Pipeline Steps

Content coming soon...

Suggested Topics

• Task dependencies: the >> operator and setupstream/setdownstream
• Running ingestion before transformation: ensuring dbt runs only after fresh data arrives
• Branching and conditional execution: BranchPythonOperator and trigger rules
• Parallel vs sequential tasks: when to run steps concurrently
• Task groups: organizing complex DAGs into logical sections
• Cross-DAG dependencies: using TriggerDagRunOperator and ExternalTaskSensor
• XCom: how Airflow passes small data between tasks (e.g., row counts, file paths, status flags)
• Error propagation: how a failed upstream task affects downstream steps
• Designing DAGs for clarity: keeping pipelines readable and maintainable

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.