Running the Pipeline Locally

Your team inherits a working starter pipeline. Before this chapter, read the three stage maps:

Ingestion, ACA, and CI/CD - Python package, Container Apps jobs, data-ci-cd.yaml
dbt models - variables, read_files, schemas, stg → marts
Publish to Postgres - which mart lands in Railway

This chapter runs those three stages by hand (Path A). Orchestrating with Airflow then runs the same chain through Astro. Promoting to Production is Mode 3–4 on the team VM after merge.

How this chapter is structured

Same pattern as chapters 11 and 12: starter first, then your source.

Beat	What you do	Jump to
1. Starter	Prove Path A on the template Arbeitnow feed. Do not change ingest or dbt yet.	Preflight → Path A
2. Your source	After a local fetch works, change ingest + dbt (+ publish if needed) using the stage chapters, then re-run Path A on your feed.	Your source on Path A

Finish beat 1 before beat 2. Do not jump to Mode 2 or prod to "try the new API."

When to run each beat

Goal	When
Beat 1 (starter)	Week 1, before you change any code
Beat 2 (your source)	Week 2, after a local Python fetch works on your laptop

All data trainees pair on the same chosen source before anyone splits ingestion vs platform in week 4. See Team Timelines.

By the end of this chapter, you should be able to:

Complete preflight checks for your personal dev environment
Run ingest, dbt, and publish by hand in the correct order (starter, then your source)
Verify that the right data arrived, not just that a command exited without an error
Point at the stage chapters for which files to change when you leave Arbeitnow

Prerequisites

Work from the data folder of your team's monorepo (see the team table below). You need:

The repository cloned and your data/.env filled in from .env.example (including Railway Postgres from Key Vault; see Secrets & Variables and One-time setup below)
az login against the HYF tenant (set AZURE_TENANT_ID in .env if you work in multiple tenants)
A personal Databricks token minted in the UI. See dbt on Databricks for the steps
uv from Week 12 setup

The cohort has three data teams: A, B, and C. Each has its own monorepo, storage account, Databricks catalog, and team resources. Substitute your team letter everywhere you see the team-a example below (a → b or c, team_a → team_b or team_c, and so on).

Team	Monorepo	Storage account	Databricks catalog
A	`HackYourFutureProjects/c55-final-project-group-A`	`sthyffpteama`	`team_a`
B	`HackYourFutureProjects/c55-final-project-group-B`	`sthyffpteamb`	`team_b`
C	`HackYourFutureProjects/c55-final-project-group-C`	`sthyffpteamc`	`team_c`

<aside> 💡 Examples use team-a resource names; team A lands under events. If you are on team B or C, swap the letter in storage and catalog names. Team A's LANDING_PATH includes the full source folder (…/events); teams B and C use a parent prefix and append /postings in staging SQL. Follow your team's .env.example for the exact path shape. Never run commands or open URLs meant for another team.

</aside>

Dev and prod use different names on purpose

Your laptop and the scheduled run share the same code but not the same targets. That split stops one person's local experiment from overwriting what the app reads.

Layer	Your dev run	The prod run
Storage container	`dev` (prefix `dev_<yourname>` or `aca-dev`)	`prod` (prefix `raw`)
Databricks schema	`dev_<yourname>`	`analytics`
Postgres schema	`analytics_dev` via `BACKEND_PG_PUBLISH_SCHEMA` (Running the Pipeline Locally matrix; safe to overwrite)	`analytics`

flowchart LR
    subgraph laptop["Your laptop / Astro"]
        L1["uv run / Mode 1–2"]
        L2["dev container<br/>personal prefix or aca-dev"]
        L3["Databricks<br/>dev_yourname"]
        L4["Postgres<br/>analytics_dev"]
        L1 --> L2 --> L3 --> L4
    end
    subgraph refused["Blocked from your laptop"]
        R1["prod container"]
        R2["Databricks analytics"]
        R3["Postgres analytics"]
    end
    L1 -.->|no write| R1
    L1 -.->|no write| R2
    L1 -.->|no write| R3

<aside> 💡 Editable deep-dive template (optional): week_15__architecture.excalidraw. Prefer mermaid in your README.

</aside>

The table and diagram split the work by where it runs. The split that catches people out is what you are allowed to write.

<aside> 💡 You write the dev storage container and your own dev_<name> schema. You cannot write the prod container or the analytics and ops schemas. Those belong to the scheduled run. Unity Catalog still has a volume path under /Volumes/team_a/landing/…; that landing segment names a Unity Catalog volume, not a storage container. Your files live in dev or prod, and the volume is how Databricks reads them.

</aside>

Your own slice of the dev container is set by LANDING_PREFIX in data/.env. Set it to your name, keep it consistent, and every command below that takes a prefix will find your files rather than a teammate's.

Two settings split write from read:

LANDING_PREFIX (+ source folder constant in pipeline.py, usually postings): where local Python ingest writes (Path A and Astro Mode 1). Not used by the ACA job in Mode 2; that job has its own env in Azure (aca-dev).
LANDING_PATH: Unity Catalog prefix where dbt reads. Teams B/C: parent prefix only; staging appends /postings in SQL (read_files('{{ var("landing_path") }}/postings', …)). Team A: full source path including /events.

For Path A they must agree (teams B/C example):

# Teams B/C: LANDING_PATH is the parent prefix only (no /postings). Staging appends /postings.
LANDING_PREFIX=your-name
SOURCE_API_URL=https://www.arbeitnow.com/api/job-board-api
LANDING_PATH=/Volumes/team_b/landing/dev/your-name

Team A copies the path shape from .env.example (full …/events path, Ticketmaster URL).

<aside> ⚠️ scripts/preflight.sh warns when LANDING_PREFIX and LANDING_PATH disagree. Path A should show LANDING_PREFIX and LANDING_PATH are aligned. In Mode 2 you only change LANDING_PATH to …/dev/aca-dev; keep your personal LANDING_PREFIX for Mode 1. A mismatch warning there is expected.

</aside>

The pipeline has three stages in order: ingest, dbt build, then publish. You run each one by hand below.

Preflight

Five minutes here saves an hour of reading logs.

One-time setup

From the repository root:

cd data

# Creates your personal .env from the tracked template (gitignored).
cp .env.example .env

Pause and fill data/.env before preflight. Two blocks matter most for Path A: your Databricks personal settings (DATABRICKS_TOKEN, DBT_SCHEMA=dev_<yourname>, LANDING_PREFIX / LANDING_PATH), and the Railway Postgres fields below.

Railway Postgres from Key Vault

Publish talks to your team's real Postgres, not a container on your laptop. Fetch these four secrets from Key Vault (team-a names below; swap team-a for team-b or team-c). Full matrix: Running the Pipeline Locally.

fp-pg-host-team-x (Railway hostname → BACKEND_PG_HOST)
fp-pg-port-team-x (TCP port → BACKEND_PG_PORT)
fp-pg-db-team-x (database name → BACKEND_PG_DB, usually project_db)
fp-pg-analytics-dev-team-x (shared password → BACKEND_PG_PASSWORD)

Key Vault has no username secret. The login name is always the fixed string analytics_dev_user (see Secrets & Variables): one shared Postgres role for the whole team.

az keyvault secret show --vault-name kv-hyf-data --name fp-pg-host-team-a --query value -o tsv
az keyvault secret show --vault-name kv-hyf-data --name fp-pg-port-team-a --query value -o tsv
az keyvault secret show --vault-name kv-hyf-data --name fp-pg-db-team-a --query value -o tsv
az keyvault secret show --vault-name kv-hyf-data --name fp-pg-analytics-dev-team-a --query value -o tsv

Put those values into data/.env like this. Username and publish schema are fixed for every teammate; do not invent personal variants:

BACKEND_PG_HOST=          # fp-pg-host-team-a
BACKEND_PG_PORT=5432      # or the port secret if it differs
BACKEND_PG_DB=project_db  # fp-pg-db-team-a
BACKEND_PG_USER=analytics_dev_user   # always this exact name - not analytics_dev_<yourname>
BACKEND_PG_PASSWORD=      # fp-pg-analytics-dev-team-a (same password for the whole team)
BACKEND_PG_PUBLISH_SCHEMA=analytics_dev
BACKEND_PG_SSLMODE=require

<aside> ⚠️ BACKEND_PG_USER is analytics_dev_user for every person on the team. A username like analytics_dev_<yourname> does not exist, so DBeaver and publish fail with an authentication error even when the password secret is correct.

</aside>

analytics_dev_user can write analytics_dev and cannot touch production analytics. The production role uses fp-pg-analytics-password-team-x (analytics_user password; Promoting to Production matrix). Only the Airflow VM may read that secret; your laptop is not granted it on purpose.

Then run preflight from the repository root:

cd ..

# Azure session, Databricks token, warehouse, Postgres, settings alignment.
# Reads data/.env itself, so nothing needs exporting first.
scripts/preflight.sh

| --- | --- | --- |

<aside> ⚠️ If preflight fails on infrastructure you did not configure (wrong tenant, missing Key Vault grant), stop and ask a teacher. Do not patch Azure roles or VM settings yourself.

</aside>

Work through Path A below in order.

<aside> ⌨️ Hands on: Run ingest, dbt, and publish by hand.

</aside>

Path A: run each step by hand

Path A (ingest, dbt, and publish by hand) is the fastest loop while you are changing code. Each step is a command.

Stage detail: ingestion, dbt, publish. Prior-week refreshers: Week 3 ingest, Week 13 dbt on Databricks.

Step 1: ingest a file into the landing path

cd data && set -a && . ./.env && set +a

uv run python -m src.ingestion.pipeline

You should see landed and rejected counts and the exact path:

landed 175 records, 1411925 bytes, to dev/<your-prefix>/postings/... on sthyffpteam<x>
Pipeline finished: 175 landed, 0 rejected, readable at /Volumes/team_<x>/landing/dev/<your-prefix>/postings

(Team A: expect events instead of postings in both lines.)