Parameterized Runs and Backfills
In Introduction to Orchestration you saw why a pipeline needs an orchestrator. This chapter gets you to a running one: local Airflow, a first DAG, and the UI views you will spend the rest of the week in. Theory about schedulers and metadata databases only makes sense once you have watched those components start up on your own machine, so this chapter is hands-on first and architecture second.
By the end of this chapter, you should be able to:
@task) wires your starter DAG's tasks together, and know that classic operators are the alternative you will use in Sequential Pipelines.Astro CLI is the Astronomer command-line tool that wraps Docker Compose around a full Airflow stack (scheduler, dag-processor, api-server, triggerer, Postgres metadata DB, with the executor/worker bundled into the scheduler container). You run one command and get the same Airflow image real teams use in production, without installing Python packages into your system environment. This is also the stack the shared class VM runs, so skills transfer directly.
<aside> 💡 Docker Compose starts several containers together as one stack from a single file, the step up from the single containers you ran in Week 5. You never write or edit that file here: Astro CLI generates and runs it for you (see Docker Compose in Week 5 if you want to see what one looks like).
</aside>
While Docker runs the containers, we need a command-line tool to orchestrate starting, stopping, and developing against that local stack.
<aside> 🤓 Curious Geek: System binary versus Python package
The Astro CLI is a standalone command-line utility written in Go, not a Python package. Because its job is to manage local Docker containers and coordinate the Airflow stack, it runs as a native system binary on your host machine (similar to git or docker).
We still use uv sync inside our project directories to install Airflow packages locally (which provides your IDE with autocomplete and code intelligence when writing DAGs), but starting and running the Airflow server and database stack requires the native astro tool.
</aside>
docker ps in your terminal, which should print a table header without error).Pick the instructions for your operating system to install the Astro CLI. You can also refer to the official Astro CLI installation guide for more options.
Run the Homebrew command below. See the official macOS installation guide for manual install options.
brew install astro
Ensure your user is in the docker group so docker ps works without sudo, then run the installation script. See the official Linux installation guide for other package managers.
curl -sSL install.astronomer.io | sudo bash -s
Run the installation script inside your WSL2 Ubuntu terminal (or Git Bash terminal in VS Code). See the official Windows installation guide for more details.
curl -sSL install.astronomer.io | sudo bash -s
<aside> 💡 Running Astro from Windows PowerShell directly works, but performance is significantly worse than inside WSL2.
</aside>
Only use this fallback if WSL2 is blocked on your machine (expect slow startup). Run in PowerShell:
winget install -e --id Astronomer.Astro
Verify the install:
astro version
You should see a version banner like Astro CLI Version: 1.41.0. Any version from 1.40 upward works for Week 12 (older versions ran Airflow 2.x; the chapter assumes Airflow 3).
<aside> 🎬 Terminal Tutorial: Astro CLI startup sequence
</aside>
https://gist.githack.com/lassebenni/e9f170434422da42f9b5dd25e0b51936/raw/week_12__astro_cli_startup_terminal.html
Create an empty folder for your Week 12 work and scaffold an Astro project inside it:
mkdir week12-airflow && cd week12-airflow
astro dev init
<aside>
⚠️ Windows: keep this project off OneDrive. Create it on a plain local path such as C:\dev\week12-airflow (or your WSL home), not inside a OneDrive-synced folder like …\OneDrive\Documents\. OneDrive keeps files as cloud placeholders, so Docker mounts an empty file and the DAGs you create never appear. Also confirm Docker Desktop shares the drive under Settings → Resources → File Sharing. If a DAG you added does not show up, see Gotcha #11.
</aside>
astro dev init writes a minimal project layout (you will tour it in a moment). Then start the stack:
astro dev start
The first start downloads the Airflow runtime image (1-2 GB) and can take 3-5 minutes. Subsequent starts take about 30 seconds.
When it finishes, Astro prints the URLs it exposed. The exact URL depends on your project directory name and a random port chosen at init:
➤ Airflow UI: <http://week12-airflow.localhost:6563>
➤ Postgres Database: postgresql://localhost:17733/postgres
➤ The default Postgres DB credentials are: postgres:postgres
Open the URL printed in your terminal (not localhost:8080): Astro uses a per-project subdomain and random port so multiple projects can run side by side. Airflow 3's local auth lets you straight into the UI; no password prompt. Copy the Airflow UI URL now; you will open it many times this week.
Astro start overview
<aside>
⚠️ First startup is slow because Docker is pulling the Airflow image. If you see "Waiting for Airflow to be healthy" for over 5 minutes, hit Ctrl+C and check Docker Desktop's resource settings: Airflow needs at least 4 GB of RAM allocated to Docker.
</aside>
Once the stack is healthy, take 30 seconds to poke around the UI before creating your own DAG.
<aside>
⌨️ Hands on: Run astro dev start, wait for Astro to print the three ➤ URL lines, and open the Airflow UI URL from your own terminal in a browser. Confirm you see the DAGs page (it will be mostly empty except for one example_astronauts DAG that Astro ships by default).
</aside>
In another terminal, confirm the five Airflow components are now running as containers:
docker ps --format "table {{.Names}}\t{{.Status}}"
NAMES STATUS
week12-airflow_c29a97-dag-processor-1 Up 4 minutes
week12-airflow_c29a97-triggerer-1 Up 4 minutes
week12-airflow_c29a97-scheduler-1 Up 4 minutes
week12-airflow_c29a97-api-server-1 Up 4 minutes
week12-airflow_c29a97-postgres-1 Up 4 minutes
You should see five containers whose names end in scheduler, triggerer, api-server, dag-processor, and postgres (the week12-airflow_c29a97 prefix is your project name plus a random per-project hash, so yours will differ). Each one maps to a piece of the architecture this chapter talks about next.
Astro's astro dev init generates a starter DAG at dags/example_astronauts.py (which contains the example_astronauts DAG). It is functional but noisy (pulls from a NASA API). Delete it and replace it with a minimal DAG so the moving parts are obvious:
rm dags/example_astronauts.py
Create dags/hello_pipeline.py:
# dags/hello_pipeline.py
from datetime import datetime
from airflow.sdk import dag, task
@dag(
schedule="@daily",
start_date=datetime(2025, 1, 1),
catchup=False,
tags=["week12", "intro"],
)
def hello_pipeline():
@task()
def ingest() -> int:
return 42
@task()
def transform(count: int) -> None:
print(f"Processed {count} rows")
transform(ingest())
hello_pipeline()
Save the file. Astro's dag-processor container parses dags/ on a schedule (roughly every 30 seconds in the default local config). New DAGs can take up to a minute to appear in the UI. If you are impatient, force an immediate re-scan:
astro dev run dags reserialize
Create dags/hello_pipeline.py with the code above and wait up to 60 seconds (or run astro dev run dags reserialize to speed it up). Then follow these steps in the Airflow UI:
Step 1: Switch to the Dags tab
Make sure you are in the Dags page of the Airflow UI. You should see both example_astronauts and hello_pipeline listed.
Airflow Dags page overview
Step 2: Manually delete the example DAG
Since the example file is not present on disk, the database still holds its metadata cache. Click the red trash can icon (🗑) on the right side of the example_astronauts row to clean it up.
Deleting example DAG from UI
Step 3: Unpause the DAG
Toggle the switch at the far left of the hello_pipeline row to unpause it.
Step 4: Trigger the DAG
Click the play button (▶) on the right side of the hello_pipeline row.
Clicking the play button to trigger DAG
Step 5: Select Trigger Options
Keep the default options and click the blue Trigger button.
Trigger options modal with default values
Step 6: Open the DAG to see the results
Click the hello_pipeline name to open its grid view. Wait for the run column cells to turn dark green, indicating all tasks succeeded.
Airflow grid view showing successful hello_pipeline run
Now spend five minutes exploring the UI around that one run. The four views you will use every day this week:
| --- | --- | --- |
To find the log output Processed 42 rows of your first run, navigate through these pages in the Airflow UI:
Step 1: Open the DAG overview page
Click on the name hello_pipeline in the DAG list to open its overview (URL: http://week12-airflow.localhost:6563/dags/hello_pipeline). Here you see the task graphs, trigger history, and metadata.
DAG overview page
Step 2: Go to the Runs list
Click the Runs tab at the top of the details panel (URL: http://week12-airflow.localhost:6563/dags/hello_pipeline/runs). This lists every execution of the DAG, showing their ID, logical date, state, and duration.
DAG runs list
Step 3: Inspect the Run details and Task Instances
Click on the run ID link (e.g., manual__2026-07-16T06:15:04.915704+00:00) in the table to drill down into the run's details (URL: http://week12-airflow.localhost:6563/dags/hello_pipeline/runs/manual__2026-07-16T06:15:04.915704+00:00). Here you see the state of every task inside that specific run.
DAG run details page
Step 4: View the Task Instance logs
Click on the transform task in the task instance list, then click the Logs tab at the top of the detail panel (URL: http://week12-airflow.localhost:6563/dags/hello_pipeline/runs/manual__2026-07-16T06:15:04.915704+00:00/tasks/transform). Scroll or search through the logs to verify the output: Processed 42 rows.
Task instance logs showing Processed 42 rows
Now the architecture discussion has something concrete to point at. Astro's local stack runs five containers that map to Airflow's six core components: the executor/worker is bundled into the scheduler container rather than running on its own. The docker ps output already showed you the five containers:
flowchart LR
dag["<b>DAG files</b><br/>dags/ folder"] --> dp["<b>DAG Processor</b><br/>parses files"]
dp --> db[("<b>Metadata DB</b><br/>Postgres, task state")]
db <--> sch["<b>Scheduler</b><br/>decides what runs"]
sch --> exec["<b>Executor / Worker</b><br/>runs the task"]
exec --> db
db <--> api["<b>API Server</b><br/>UI + REST API"]
db <--> trg["<b>Triggerer</b><br/>async sensors"]
…-dag-processor) parses every file in dags/ on a schedule and writes the serialized DAG (its task list, dependencies, and schedules converted into a database-friendly JSON representation) into the metadata DB. In Airflow 2 this lived inside the scheduler; Airflow 3 split it out so parsing crashes cannot take the scheduler down.…-scheduler) reads serialized DAGs from the DB and queues tasks that are due to run.