Week 12 - Orchestration

Welcome to Week 12! You have built pipelines that ingest, transform, and model data. Now it's time to make them run automatically and reliably. This week covers orchestration: scheduling your pipelines, chaining steps together, handling failures, and monitoring production runs.

By the end of this week, you will have a fully orchestrated pipeline that runs on a schedule, chains ingestion and transformation steps in the correct order, supports parameterized runs and backfills, and surfaces errors through Airflow's logs and the UI.

How this week helps you get hired

Junior data engineers in the Netherlands are usually trusted with existing pipelines before they build new ones. Teams expect you to:

understand why a pipeline failed and what to do about it
backfill historical dates safely without creating duplicates
communicate incidents clearly while a run is broken
avoid silent data corruption when reruns happen

Week 12 trains exactly these skills. This is strong portfolio evidence because it shows operational ownership: not just writing code that works once, but running it reliably every morning.

Learning goals

Understand what orchestration means and why it is essential for production data pipelines
Set up and navigate Apache Airflow including DAGs, tasks, operators, and the web UI
Define cron-based schedules and event-driven triggers to automate pipeline runs
Chain pipeline steps sequentially so ingestion runs before dbt transformation
Use parameterized runs and backfills to reprocess data for specific dates or date ranges
Monitor pipeline health and debug failed runs using Airflow logs
Develop DAGs locally with the Astro CLI and deploy them to the class's shared Airflow via a shared Git repo
Work alongside classmates on a single shared scheduler without interfering with each other's pipelines

By the end of this week, you should be able to explain

how to trigger and debug a failed DAG run in Airflow
why ds (logical date) matters for reruns
how your DAG prevents duplicate processing during backfills

Chapters

Supplementary

Glossary: Week 12

Career relevance: Week 12

Going Further

History of Data Orchestration

Slides (PDF)