Optional: Enriching with an LLM
Two kinds of failure show up in the final project: pipeline traps you hit while running the platform, and project traps you hit while building your own features on top. This page collects both.
These come from running the starter pipeline. Full walkthroughs are in Running the Pipeline Locally, Orchestrating with Airflow, and Promoting to Production.
The error often reads http-code=401, method=OpenSession, which sounds like connectivity. Generate a new token in the Databricks UI and update data/.env. Token steps are in Week 13: dbt on Databricks.
max_active_runs=1 can make your trigger look hungUnpausing the DAG releases the most recent scheduled interval straight away, and that run takes the only slot, so your manual trigger sits in queued. It is not hung. Let the first run finish or mark it failed, then trigger again. The DAG sets catchup=False, so you get that one run rather than every interval since start_date. See Week 12: Scheduling and Triggers for max_active_runs and queued runs.
Otherwise your laptop's Astro stack can start container jobs and wake the warehouse on the prod schedule while you are not watching.
read_mart raises on zero rows because publishing an empty mart over a full one is worse than failing. A publish failure usually means dbt did not build the expected data. Diagnose at the dbt end, not in Postgres.
Laptop publish and DBeaver use analytics_dev_user for everyone on the team, with the shared password in fp-pg-analytics-dev-team-x (Running the Pipeline Locally matrix). There is no per-trainee Postgres user. If you set BACKEND_PG_USER to something like analytics_dev_<yourname>, authentication fails even when the Key Vault password is right. Fix: use the exact string analytics_dev_user. Setup steps are in Running the Pipeline Locally.
The team VM login is student-<your-name>, where <your-name> matches the suffix in fp-airflow-password-<your-name>. It is not airflow, not your GitHub handle, and not analytics_dev_user. Full table: Airflow UI login.
Re-running ingestion the same day replaces that day's file rather than appending. A second run does not double your row count.
Astro assigns a local port per project. After astro dev restart, use the URL it prints, or run astro dev ps from data/airflow. A bookmarked URL from an old session can silently point at the wrong instance. Stop stray stacks with astro dev stop if you have more than one Astro project open.
If you skip building and pushing before triggering with INGEST_MODE=aca, a green ingest task only proves the previous image still works, not your current code. Build and point the job at your image first: Orchestrating with Airflow.
If the prod baseline from Promoting to Production does not go green, or Variables render empty in prod Airflow, stop and ask a teacher. Do not patch Azure RBAC, VM compose, or Key Vault permissions yourself.
landing vs prod in old docsProduction ingestion writes the prod storage container (LANDING_CONTAINER=prod in CI). Development uses the dev container. Some older README or video material may say landing; if commands fail with a missing container, check the name in your repo's workflow and .env, not the recording.
The most common support question. Check in order:
| Check | Mode 1–2 (laptop) | Mode 3 (VM-dev) | Mode 4 (prod) |
|---|---|---|---|
| Postgres schema | analytics_dev |
analytics_dev |
analytics |
| Databricks schema | dev_<yourname> |
dev_airflow |
analytics |
| Storage container | dev + your prefix (Mode 1) or aca-dev (Mode 2) |
dev + aca-dev |
prod + raw |
| Storage account (team-a example) | sthyffpteama |
same | same |
| Ingestion job | Mode 1: none · Mode 2: job-fp-ingest-dev |
job-fp-ingest-dev |
job-fp-ingest |
| Who runs dbt | your personal token | team service principal | team service principal |
Also verify: INGEST_MODE and LANDING_PATH match the path your ingest actually wrote; prod ACA job image tag matches your merge SHA; job SOURCE_API_URL is your source (not the starter); you waited for a DAG run after merge, not just for CI to finish.
data/.env and ingest.py on main do not set the Container Apps job URL. Mode 2 and prod only start the job; the container reads SOURCE_API_URL from job env that deploy-prod / deploy-dev rewrite on every merge. You do not create a second ACA job for the new API.
If logs still show arbeitnow.com (or another starter URL) after you merged a new source:
.github/workflows/data-ci-cd.yaml and confirm the workflow env: block and both deploy jobs' env_vars= use your URL as literals (not a forgotten GitHub Actions Variable).deploy-prod / deploy-dev steps succeeded.Fix the workflow in a PR. Do not rely on a one-off az containerapp job update from a laptop: the next data merge will overwrite job env again.
stg_postings.sql reads whatever LANDING_PATH points at (teams B/C: plus /postings in SQL; team A: full …/events path), not where ACA wrote by default. Mode 2 lands under dev/aca-dev/postings/… from the job config. If .env still has your personal path, dbt builds from stale or empty files and field mappings look broken for hours.
A green ingest task is not proof dbt will read those bytes. Compare the path in the ingest task log with LANDING_PATH before rewriting staging SQL.
Fix: set LANDING_PATH=/Volumes/team_<x>/landing/dev/aca-dev (parent only; staging adds /postings). Keep LANDING_PREFIX=your-name for Mode 1; do not set LANDING_PREFIX=aca-dev unless you mean local ingest to write the shared team folder. Restore your personal parent LANDING_PATH when you switch back to Mode 1. See Mode 2.
After you swap APIs, staging still runs read_files on the whole landing folder. It does not pick "the latest file only". Old starter JSON (Arbeitnow) next to the new shape (different date formats, fields, or types) makes casts fail, often on a unix created_at that looks like a random integer.
That is not "dbt stuck on the first file"; it is reading every blob and choking on incompatible ones. Fix: delete or move obsolete files under that folder (prod raw/postings and any dev/…/postings you still use; team A: …/events), or change the SOURCE_NAME constant in pipeline.py and the matching staging read_files path, then clean the old folder. Prefer keeping postings (B/C) or events (A) and cleaning the folder. If Unity Catalog only lets you see metadata, ask a teacher. Students should not invent a workaround job.
dbt build can be green on jobs (or another new mart) while src/publishing/sync.py still reads fct_postings_enriched and writes fct_postings. The DAG and Path A python -m src.publishing.sync use those defaults unless you change them (or pass --mart / --table).
Fix in the same PR as the rename: update sync defaults, the DAG publish task args if any, and the mart contract with backend. Re-run Path A and check analytics_dev for the new table and columns before merging. See Publish (only if the mart contract moved).
The browser hangs on /dags while tasks may still run. Usually a stuck apiserver with leaked Postgres transactions, not a VM capacity problem. Students should ask a teacher; do not edit VM compose yourself. Teachers: see Teacher onboarding (watchdog script + service restart).
AuthorizationPermissionMismatch on blob uploadIf INGEST_MODE=local and ingest fails uploading to the dev container, check AZURE_CLIENT_ID / AZURE_CLIENT_SECRET are in data/.env (fetch from Key Vault; do not remove them). Astro runs ingest inside Docker and uses that dev-run service principal for blob writes in Mode 1 and for starting the dev job in Mode 2. Do not remove those vars to "fix" a 403: without them every credential chain fails in the worker.
A 403 with the vars present usually means the platform grant is still propagating (wait a few minutes) or you are pointed at the wrong container. You should be writing dev, not prod. Ask a teacher if it persists after astro dev restart.
Many sites block cloud IP ranges. If the same script works on your laptop and returns 403 or empty responses from a Container Apps job, the source may be blocking Azure. Prefer a proper API; do not use proxies or fresh accounts to evade bans.
If the Aug 13 recording disagrees with these pages or your repo README, trust the written material.
These show up while building your own source, models, and syncs on top of the starter platform.
Trying to build too much and delivering nothing complete. Cut in the order in Requirements and Deliverables, and say what you cut in the README.
A project without a README, data dictionary, and architecture diagram is hard to evaluate and impossible to hand off. The diagram is a week-1 artefact you revise, not a week-5 recreation from memory.
Waiting until the last week to run on Azure usually ends in a demo that breaks live. The starter pipeline is already deployed. Prove it green in week 1, then extend it in week 2.
Building a pipeline that outputs data in a shape the backend cannot read wastes everyone's time. Agree the mart contract in week 1.
Publishing a star of tables (companies, jobs, skills, …) and expecting the backend to join it per endpoint pushes join logic onto another team. See Cross-Track Collaboration: Mart shape.
Letting the backend copy your wide mart into normalized app tables duplicates data and refresh logic for no capstone-scale gain. Prefer direct reads, indexes, or scheduled caches.
One team member holds all the knowledge and the project stalls when they are unavailable. The tool switch in Team Timelines exists so every DE on your team can explain the others' work at the demo.
A published mart with no tests on its keys is the most common way a demo breaks in front of people. Tests on the mart the backend reads are not optional.
Your mentor may pick a merged PR at random and ask the reviewer to explain the change. Approving without reading is visible faster than you would think.