Introduction to Orchestration

Airflow Fundamentals

Scheduling and Triggers

Sequential Pipeline Steps

Parameterized Runs and Backfills

Monitoring and Debugging

Practice

Assignment

Gotchas & Pitfalls

Week 12 Lesson Plan (Teachers)

Airflow Fundamentals

Content coming soon...

Suggested Topics

• What is Apache Airflow and why it is the industry standard for data orchestration
• Core architecture: scheduler, webserver, executor, metadata database
• DAGs (Directed Acyclic Graphs): defining pipelines as code in Python
• Tasks and operators: BashOperator, PythonOperator, and custom operators
• The Airflow web UI: navigating DAG views, task logs, and run history
• Astronomer platform overview: deploying DAGs, the Astronomer CLI (astro dev init, astro deploy), and the Astronomer UI
• Local development with Astronomer CLI: astro dev start to run Airflow locally before deploying
• Connections and variables: managing credentials and configuration
• TaskFlow API (@task decorator): the modern Airflow 2.x way to define tasks, shown alongside the classic operator approach
• Writing your first DAG: a simple two-task pipeline

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.