Parameterized Runs and Backfills
A single place to look up every Airflow and orchestration term you meet this week. Entries are ordered by the chapter that first introduces the term: each term lives in exactly one place, and each chapter back-links to its glossary entry on first use. Skim the section for the chapter you are reading; the section above it covers terms you have already met.
Coordinating pipeline tasks, dependencies, retries, and run history across a full workflow. What Airflow does. Compare with scheduling, which only answers "when to start."
An open-source workflow management platform used to programmatically author, schedule, and monitor data pipelines as Directed Acyclic Graphs (DAGs).
The end-to-end sequence of data steps you want to run together. In Week 12 the pipeline is ingest → load → dbt run → dbt test.
The step-by-step map of your pipeline expressed as a Python file in dags/. Acyclic means no loops: step B can depend on step A, but A cannot also depend on B.
One unit of work inside a DAG, usually a Python function decorated with @task. Deeper coverage in Airflow Fundamentals.
The order relation between tasks, written as a >> b or implied by passing one task's return value into another. Detailed in Sequential Pipeline Steps.
Automatic re-attempt of a task after a temporary failure, configured via retries=N and retry_delay. Tested in Testing DAGs.
Rerunning the same task or DAG gives the same correct result without duplicates. Usually achieved with a DELETE-by-partition-key before the INSERT. Drilled in Sequential Pipeline Steps and the assignment.
One execution of a task for one DAG run. The same task can have many task instances across runs (one per scheduled date, plus retries).
A pre-built task class such as BashOperator or PythonOperator. The @task decorator is sugar over PythonOperator.
The decorator-based way of writing DAGs (@dag, @task). Pairs naturally with XCom for passing values.
The Python import path you see in every Week 12 DAG (from airflow.sdk import dag, task). This is Airflow 3's public API surface; the older airflow.decorators is 2.x-era.
The command-line tool provided by Astronomer to initialize, develop, test, run, and deploy Apache Airflow environments locally inside Docker containers.
A tool for defining and running multi-container Docker applications. It uses a YAML file to configure and start all of the application's services (scheduler, database, API server, etc.) together.
A compatibility layer in Windows 10 and 11 that allows you to run a full Linux environment (usually Ubuntu) directly on Windows. This lets you run native Linux tools and commands without the overhead of a virtual machine.
The Airflow service that decides which task instances are ready to run and hands them to workers. Distinct from "scheduler" in the generic sense (a thing that runs commands on a clock), which is only one of orchestration's responsibilities.
The Airflow 3 service that hosts the web UI and REST API. Renamed from webserver (its name throughout Airflow 2.x) in 3.0. Detailed in Deploying to Shared Airflow.
The separate Airflow 3 service that parses .py files in dags/ into the scheduler's DAG list. A broken DAG fails parsing here without crashing the scheduler for everyone. Tested in Testing DAGs.
The component (inside the scheduler) that determines how tasks are run (e.g. locally in parallel, via a queue of workers, or in Kubernetes pods).
The process that actually executes the task code. In Astro CLI's local stack, workers and scheduler share a container.
The Airflow service that runs async-style sensors and deferrable operators without holding worker slots. Sits next to scheduler and dag-processor in the Astro CLI's local stack.
The Postgres database Airflow itself uses to store DAG definitions, run history, task states, and XComs. Not to be confused with the Azure Postgres where your data lives.
(cross-communication) Airflow's built-in mechanism for passing small values (row counts, paths, IDs) between tasks via the metadata DB. Good for int/str; bad for DataFrames or filesystem paths.
"Upstream of B" means "runs before B" (A in A >> B). A failure upstream puts the downstream task into upstream_failed. Drilled in Monitoring and Debugging.
The web interface (the api-server) where you trigger DAGs, view logs, and inspect task states. Key views: DAG list, Grid view, Graph view, Task instance → Logs, Rendered Templates tab.
The primary monitoring interface in the Airflow UI. It displays DAG runs as vertical columns and task states as colored grid cells, making it easy to see historical task-instance states, logs, and execution times at a glance.
The visualization interface in the Airflow UI that renders tasks as nodes and dependencies as directed arrows. It shows the layout (DAG structure) of the pipeline and helps verify that the task execution order is correct.
The cadence at which Airflow should create DAG runs. Accepts cron strings ("0 6 * * *"), presets ("@daily"), or cron-based timetables.
A time-based job scheduler in Unix-like operating systems, or the five-field syntax min hour day month weekday (also called a cron expression) that Airflow's schedule= accepts to run DAGs. Inherited from Paul Vixie's 1987 cron rewrite; read the optional history chapter for the story.
Convenience aliases like "@daily", "@hourly", "@monthly" that expand to standard cron strings. Use them for clarity unless the cadence is non-standard.
ds)The date label for the data interval being processed. Written {{ ds }} in templates. Stable across retries and manual reruns: this is what makes backfills idempotent. Detailed in Parameterized Runs and Backfills.
The [start, end) window of data a run is responsible for. For a daily schedule the interval for 2024-03-10 runs after 2024-03-10 is complete.
A named slice of data split by date or time period (for example one day's trips, one month's parquet file, or rows where pickup_date = '2024-01-15'). Pipelines use the logical date as the partition key so each scheduled run always reads or writes the same slice, even if the run starts late.
When set to True, Airflow creates historical runs from the DAG's start_date up to now. False means "only the next scheduled interval." Default is False in this course; flip to True only when you want automatic backfilling on un-pause.
Starting a DAG run from the UI or CLI outside its schedule. On Airflow 3, manually-triggered runs may have logical_date=None, so code that reads ds should fall back to dag_run.run_after via the _ds_from_context() helper.
The Airflow UI view that shows scheduled runs as dots on a month grid, making it easy to sanity-check whether a cron expression lands on the days you intended.
Starting a DAG or task run immediately in response to an external event (such as a database change, a webhook payload, or a new file landing in a bucket) rather than waiting for a predefined time schedule.
A task type that waits for an external condition (a file, a time, an API response) before letting the DAG continue. Use poke_interval and mode="reschedule" so sensors do not hold worker slots.
Airflow 3's pattern of triggering one DAG when another DAG produces a Dataset. Sits next to cron schedules and sensors as a third kind of trigger. Not used this week; see Going Further for context.
An Airflow object that stores a host, credentials, and metadata under a named ID (for example azure_pg). Tasks reference connections by ID rather than reading environment variables directly. Detailed in Monitoring and Debugging.
The Python client that uses a Connection to talk to a system. PostgresHook("azure_pg") is the one you use most this week.
AIRFLOW_STUDENT / airflow_<name>The per-student schema-isolation pattern. Your DAG's STUDENT constant reads this env var (or falls back to the DAG's subdirectory name on the shared VM) and writes to an airflow_<your-name> schema in Azure Postgres so you do not clobber classmates' data. Used again in Deploying to Shared Airflow.
The convention of writing to airflow_<your-name> instead of a shared analytics schema during development. Pairs with AIRFLOW_STUDENT.
Running a DAG where the execution is customized at runtime by passing dynamic parameters (via the UI, CLI, or API) to control the inputs, date ranges, or behavior of the tasks.
Running historical dates after the fact, typically with astro dev run backfill create. Airflow 3 uses deterministic backfill__<logical-date> run IDs so repeat backfills are idempotent.
A named key/value pair stored in Airflow's metadata DB. Use sparingly; prefer Connections for secrets and params for per-run values.