Publish to Postgres

Publish copies a finished Databricks mart into the backend's Railway Postgres so the app can query analytics without touching your warehouse. It is a Python task (data/src/publishing/sync.py), not a dbt model - dbt cannot open Databricks and Postgres in one model. Why: Project Architecture.

This page is short on purpose: pick which mart/table to sync, know which Postgres schema you write, then run the command in Path A. Deep swap / staging-table design lives in Optional: Inbound Sync.

By the end of this chapter, you should be able to:

Name the default warehouse mart and Postgres table the starter publishes
Say which Postgres schema Path A uses (analytics_dev) vs Mode 4 (analytics)
Know when you must edit sync.py (or CLI flags) after a source swap

What publish reads and writes

Outbound sync is not "whatever dbt built last." Defaults in data/src/publishing/sync.py:

Direction	Starter default
Read from Databricks (`DBT_SCHEMA`)	`fct_postings_enriched`
Write to Postgres (`BACKEND_PG_PUBLISH_SCHEMA`)	`fct_postings`

The two names are intentional: publish can read an enriched mart and still give the backend one stable table name.

Path A and Astro write analytics_dev (shared by the whole team - last successful publish wins). Mode 4 writes analytics. Credentials and schema keys: Secrets & Variables.

flowchart LR
    mart["Databricks mart<br/>e.g. fct_postings_enriched"]
    sync["src.publishing.sync"]
    pg["Railway Postgres<br/>analytics_dev or analytics"]
    mart --> sync --> pg

The script refuses to overwrite a populated Postgres table with an empty mart (refusing to publish an empty mart). Fix dbt first.

https://gist.githack.com/lassebenni/fd10d46f11d08119abcdcbc08a0ae6e8/raw/week_15__publish_swap_code_explainer.html

Selecting the mart (when the contract moved)

If you only changed columns inside the same table names, leave sync defaults alone. Agree the new columns with backend (mart contract).

If you renamed the warehouse mart or the Postgres table:

Change the defaults in sync.py, or pass --mart / --table when you invoke the module (see your repo's CLI help).
Do that in the same PR as the dbt rename.
Prove analytics_dev.<table> shows the new shape on Path A before Astro or prod.

A green dbt build with an unchanged sync still publishes the old names (or fails).

<aside> ⌨️ Hands on: Open data/src/publishing/sync.py and find the default mart and table names. Note them next to your planned backend table name. Do not edit until your mart contract needs a rename.

</aside>

Ready for Path A when

Running the Pipeline Locally runs ingest → dbt → publish in order.

[ ] You know publish reads a named mart, not "the last model in the project"
[ ] You know Path A publishes to analytics_dev, not analytics
[ ] You know when a rename forces a sync.py change in the same PR

Extra reading

Next up: Running the Pipeline Locally - bring all three stages together by hand.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 https://hackyourfuture.net/

CC BY-NC-SA 4.0 Icons

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.