Optional: Enriching with an LLM
Reference page, like Gotchas & Pitfalls: a lookup you open when you need a setting name or Key Vault secret, not a chapter to read front to back. Stage maps: Ingestion, dbt, Publish. Walkthroughs: Running the Pipeline Locally, Orchestrating with Airflow, and Promoting to Production.
Open this page when:
data/.env before your first Path A runAZURE_CLIENT_* before your first DAG run (Mode 1 blob writes and Mode 2 job start)| Jump to | Section |
|---|---|
| Path A: source and landing | Source and landing |
| Path A: Databricks and dbt | Databricks and dbt |
| Path A: Postgres publish | Railway Postgres (publish) |
| Astro: dev-run credential | Astro dev-run credential |
| Astro: .env vs yaml | Local Astro: .env vs airflow_settings.yaml |
| Astro / Mode 2 | Mode 2 extras |
| Prod: Airflow UI (you fetch) | Airflow UI login |
| Slack / LiteLLM | Optional |
| Teachers only | TEACHERS ONLY |
| Column | Meaning |
|---|---|
| Name | .env key, Airflow Variable, or Key Vault secret name when there is no config key (e.g. Airflow UI login) |
| Type | plaintext (type the value) or secret (fetch from Key Vault or another vault) |
| Where | Where the pipeline reads it (data/.env, team VM, ACA job) |
| Value / source | Literal to type, or Key Vault secret name (with team-a portal link when you fetch it) |
| Purpose | What it controls |
Key Vault secret names appear once: in Value / source on the row for the variable that holds the fetched value, or as Name when there is no config key (for example your personal Airflow UI password).
Swap team-x for your team letter (team-a, team-b, team-c). Examples below use team-a portal links.
Fetch any Key Vault value from kv-hyf-data:
az keyvault secret show --vault-name kv-hyf-data --name <secret-name> --query value -o tsv
<aside>
⚠️ Forbidden / ForbiddenByRbac: sign in to the HYF tenant, confirm the secret name uses your team letter, and wait a few minutes after RBAC changes.
</aside>
Running the Pipeline Locally: Path A by hand into analytics_dev. Everything in these tables goes in data/.env unless the Where column says otherwise.
| Name | Type | Where | Value / source | Purpose |
|---|---|---|---|---|
SOURCE_API_URL |
plaintext | data/.env and ACA job env (via data-ci-cd.yaml literals) |
Your API endpoint URL | HTTP target for ingestion. Laptop reads .env; Mode 2 / prod read the Container Apps job env set on merge |
STORAGE_ACCOUNT |
plaintext | data/.env |
sthyffpteam<x> |
Team blob account (sthyffpteama, …) |
LANDING_CONTAINER |
plaintext | data/.env |
dev |
Never prod on your laptop |
LANDING_PREFIX |
plaintext | data/.env |
your-name |
Where local Python ingest writes; must match the last segment of the parent LANDING_PATH on Path A (teams B/C) |
LANDING_PATH |
plaintext | data/.env |
/Volumes/team_<x>/landing/dev/your-name |
dbt read_files prefix. Teams B/C: parent only; staging appends /postings in SQL. Team A: full source folder (…/events); see callout below |
The landing subfolder (postings or events) is a constant in data/src/ingestion/pipeline.py, not a .env key or ACA job env var. Prefer keeping postings when you swap APIs (teams B/C). To rename the folder, change that constant and the matching suffix in your staging model's read_files path in the same PR.
<aside>
💡 Team A uses events, not postings. Team A's repo lands under …/events, sets LANDING_PATH to the full source path (for example /Volumes/team_a/landing/dev/your-name/events), and staging reads that path directly. Teams B and C keep the parent-prefix pattern above. Copy paths from your team's .env.example, not from another team's chapter examples.
</aside>
SOURCE_API_URL on your laptop does not change what the ACA jobs fetch. deploy-prod and deploy-dev in .github/workflows/data-ci-cd.yaml set job env on every merge from literals in the workflow file (reviewable in the PR). You do not need a new ACA job when you swap APIs. Workflow map: Ingestion, ACA, and CI/CD. Path A swap: Your source on Path A. ACA + VM verify: When you swap the source API.
Team-specific API keys (e.g. TICKETMASTER_API_KEY, GITHUB_TOKEN) are not in the tables above. Put the value in a GitHub Actions secret (Settings → Secrets and variables → Actions; write access on the repo is enough). Wire secrets.YOUR_KEY into both deploy jobs and set the ACA env with secretref:… in data-ci-cd.yaml. Keep a blank placeholder in data/.env.example for local runs. Never commit the key.
| Name | Type | Where | Value / source | Purpose |
|---|---|---|---|---|
DATABRICKS_HOST |
plaintext | data/.env |
Shared workspace URL (https://adb-….azuredatabricks.net) |
Databricks SQL endpoint |
DATABRICKS_HTTP_PATH |
plaintext | data/.env |
Shared warehouse path (/sql/1.0/warehouses/…) |
SQL warehouse HTTP path |
DATABRICKS_CATALOG |
plaintext | data/.env |
team_<x> (team_a, team_b, team_c) |
Unity Catalog for your team |
DATABRICKS_TOKEN |
secret | data/.env |
Personal PAT from Databricks UI | Signs in as you for local dbt/publish |
DBT_SCHEMA |
plaintext | data/.env |
dev_<yourname> |
Databricks schema for local builds |
DBT_TARGET |
plaintext | data/.env |
dev (rarely set by hand) |
dbt profile target; DAG picks dev when token set |
AZURE_TENANT_ID |
plaintext | data/.env |
07a14c4e-d88c-42f7-83b3-13af7e57ff3d |
HYF Entra tenant for az login |
| Name | Type | Where | Value / source | Purpose |
|---|---|---|---|---|
BACKEND_PG_HOST |
secret | data/.env |
fp-pg-host-team-x | Railway hostname |
BACKEND_PG_PORT |
secret | data/.env |
fp-pg-port-team-x | Railway TCP port |
BACKEND_PG_DB |
secret | data/.env |
fp-pg-db-team-x | Database name (usually project_db) |
BACKEND_PG_USER |
plaintext | data/.env |
analytics_dev_user |
Shared Postgres login for the whole team; not in Key Vault |
BACKEND_PG_PASSWORD |
secret | data/.env |
fp-pg-analytics-dev-team-x | Password for analytics_dev_user |
BACKEND_PG_PUBLISH_SCHEMA |
plaintext | data/.env |
analytics_dev |
Postgres schema sync.run() writes on your laptop |
BACKEND_PG_SSLMODE |
plaintext | data/.env |
require (Railway) |
TLS mode for Postgres |
Orchestrating with Airflow: Astro DAG on your laptop.
.env vs airflow_settings.yaml| --- | --- | --- |
Path A and Astro share data/.env. Keep credentials and resource names there, not in yaml. The DAG reads secrets from the environment (then Key Vault on the VM), so duplicating keys into Airflow Variables locally would not replace .env and would drift from Path A.
INGEST_MODE alone uses yaml because it is the mode switch (same idea as prod Variables), it is not in .env.example, and Astro re-applies the file on restart. Without copying airflow_settings.yaml.example, nothing sets INGEST_MODE=local and the DAG defaults to aca (Mode 2). Setup steps: One-time Astro setup.
Mode 1 (INGEST_MODE=local): reuses the Path A tables under Running the Pipeline Locally. Astro mounts data/.env into workers. The same dev-run service principal (AZURE_CLIENT_ID / AZURE_CLIENT_SECRET) uploads to the dev container in Mode 1 and starts job-fp-ingest-dev in Mode 2. Keep both vars in .env for either mode.
Mode 2 (INGEST_MODE=aca): adds the Mode 2 extras rows below. Set INGEST_MODE in yaml (not .env), change LANDING_PATH in .env so dbt reads what the dev ACA job wrote: teams B/C set the parent …/dev/aca-dev (staging adds /postings); team A set …/dev/aca-dev/events. The job lands under aca-dev from its own Azure env, not from your .env LANDING_PREFIX. Inside the ACA job, ingest uploads with the job managed identity, not the dev-run SP.
Add before your first Astro run (Mode 1 and Mode 2). Path A on the host can use az login instead; Astro workers cannot.
| --- | --- | --- | --- | --- |
| --- | --- | --- | --- | --- |
Promoting to Production: scheduled run into analytics on the team Airflow VM. Fetch your personal Airflow UI password from Key Vault; type the username below. Team VM settings and secrets are provisioned by teachers; see TEACHERS ONLY if you need the names that appear in logs. Do not copy them into data/.env.
<your-name> is the suffix in your Key Vault secret name: fp-airflow-password-<your-name> (for example fp-airflow-password-pavel-tisner → username student-pavel-tisner). It is not your GitHub handle unless teachers set it up that way. You can read only your own secret.
| --- | --- | --- | --- | --- |
<aside>
⚠️ Your laptop is not granted fp-pg-analytics-password-team-x. A local run must not overwrite analytics.
</aside>
Not required for Path A, Mode 1, or the demo.
| --- | --- | --- | --- | --- |