Introduction to Orchestration

Airflow Fundamentals

Scheduling and Triggers

Sequential Pipeline Steps

Parameterized Runs and Backfills

Monitoring and Debugging

Practice

Assignment

Gotchas & Pitfalls

Week 12 Lesson Plan (Teachers)

Introduction to Orchestration

Content coming soon...

Suggested Topics

• What is orchestration: coordinating multiple pipeline steps to run in the right order at the right time
• Why orchestration matters: manual runs do not scale, humans forget, and errors go unnoticed
• Orchestration vs scheduling: scheduling triggers execution, orchestration manages dependencies and state
• Common orchestration tools: Apache Airflow, Azure Data Factory, Prefect, Dagster
• Key concepts: DAGs, tasks, dependencies, retries, idempotency
• The orchestration layer in a modern data stack: ingestion, transformation, testing, delivery
• Why managed Airflow (Astronomer) over self-hosted: reduced ops burden, focus on DAG development
• When you do not need orchestration: small, single-step pipelines that rarely change
• Real-world example: a daily pipeline that ingests data, runs dbt, and triggers a dashboard refresh

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.