Scheduling dbt Jobs

Databricks is not only notebooks with dbt bolted on. It is a multifaceted platform: catalogs and tables, clusters for PySpark, warehouses for SQL and dbt, and Jobs under Workflows. Jobs let work run on a schedule inside the same workspace.

If you only ever click Run in a notebook or on your laptop CLI, you have used half the product.

In dbt on Databricks you proved incremental on the CLI: dbt build --select fct_trips twice, with the second run much faster. That is the right way to learn the model. Production still needs the same build to fire without you opening a laptop.

Week 12 solved cross-system orchestration with Airflow. On Databricks, the platform-native answer is a Job (a named workflow with tasks, retries, and a schedule). This chapter schedules your dbt project as a dbt task (a Job task that runs dbt CLI commands against a SQL warehouse) on hyf-dbt-warehouse.

By the end of this chapter, you should be able to:

Explain why scheduling sits inside Databricks (same platform as Catalog, Compute, and SQL), not only in an external orchestrator
Create a Job with a dbt task configured with Git provider source and pointing at hyf-dbt-warehouse
Run the Job once manually and confirm a successful run
Attach a schedule without leaving it firing on the shared class bill
Contrast Databricks Jobs with the Week 12 Airflow pattern (when each fits)

Why schedule here (on the platform)

Three options exist for running dbt models, each filling a specific role in a data stack:

Option	How it runs	Advantages	Weaknesses	Best used for
Local / Laptop CLI	Manually executed from your local terminal	Fast feedback loop while writing code, instant log output, zero platform setup	Requires human intervention, dependent on local machine state and open laptop	Development, debugging, and testing new models
Databricks Job (dbt task)	Scheduled or triggered directly inside the workspace	Native integration, automatic authentication, zero extra tools, serverless SQL warehouse compute	Scoped to Databricks ecosystem, complex cross-system pipelines require custom API calls	Scheduled dbt builds and lakehouse-native workflows
Airflow DAG (Week 12)	Scheduled DAG in an external orchestrator	Coordinates multi-system pipelines across databases, storage, Databricks, and alerts	Extra infrastructure to maintain, requires external credential and connection management	Complex enterprise pipelines spanning multiple systems

You are not replacing Airflow. You are learning that Workflows / Jobs is a first-class part of Databricks, the same way Catalog and SQL are. Many lakehouse teams use Jobs for Databricks-only work and keep Airflow (or similar) when the pipeline spans blob, Postgres, and Databricks together.

🎵 Supermode - Tell Me Why: a dbt task in Databricks Jobs runs against the SQL warehouse rather than a PySpark cluster?

<aside> 📘 Recap from Week 12: A schedule without a human in the loop is the difference between a script and a pipeline. Same idea here: the Job is the Databricks-native place that schedule lives when the work is already on the lakehouse.

</aside>

Jobs, tasks, and the dbt task type

A Job has a name, optional schedule, and one or more tasks. Task types include notebooks, Python scripts, SQL, and dbt.

For dbt, Databricks runs the dbt CLI on job compute and sends the SQL dbt generates to a SQL warehouse. That matches dbt on Databricks: models still hit hyf-dbt-warehouse, not the PySpark cluster.

flowchart TB
    sched["Schedule or Run now"]
    job["Databricks Job<br/>dbt task"]
    git["Git repo + branch<br/>(dbt project source)"]
    cli["dbt CLI on job compute<br/>deps → build"]
    wh["Serverless SQL warehouse<br/>hyf-dbt-warehouse"]
    uc["Unity Catalog table<br/>hyf.dev_yourname.fct_trips"]

    sched -->|"triggers"| job
    job -->|"fetches project at run time"| git
    job -->|"runs"| cli
    cli -->|"sends compiled SQL"| wh
    wh -->|"writes / merges rows"| uc

    classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
    class sched,job,git,cli,wh,uc box;

Read the diagram top to bottom: a schedule (or Run now) starts the Job; the Job fetches your dbt project from Git; the dbt CLI compiles and runs dbt build; the SQL runs on the serverless warehouse; the result lands as a table in Unity Catalog under your schema.

Source of the project. The dbt task needs access to your dbt project directory. Two common choices exist:

Git provider (Recommended & Production-Ready) - Point the task directly to a Git repository and branch. The workspace fetches the latest code directly from GitHub at run time. Because our class repository is public, no SSH keys or personal Git credentials are required.
Workspace - Manually upload or sync project files into a folder under your Databricks user folder.

In this chapter, you will use Git provider pointing directly to the public reference repository (https://github.com/lassebenni/nyc-taxi-dbt-reference.git, branch week-13-ch-4-dbt-solution).

🎵 Supermode - Tell Me Why: sourcing dbt projects via Git provider in Databricks Jobs is superior to dragging and dropping local files into workspace folders?

<aside> 🤓 Curious Geek: Lakeflow Jobs

Databricks has renamed and regrouped orchestration UI over time (Jobs, Workflows, Lakeflow Jobs). The skill is the same: a scheduled graph of tasks with run history. If your sidebar says Workflows instead of Jobs, open it - the Create job / dbt task flow is what this chapter means.

</aside>

⌨️ Hands on: create the dbt Job

<aside> 📘 Recap from Chapter 4: On the local CLI, you executed dbt build --select fct_trips using your .env token. Inside a Databricks Job task, Databricks runs those exact same CLI commands (dbt deps then dbt build --select fct_trips) on job compute while sending the resulting SQL queries to hyf-dbt-warehouse.

</aside>

Step 1: In the left sidebar, navigate to Jobs & Pipelines (or Workflows / Jobs).

Databricks Jobs & Pipelines list showing dev_lasse_fct_trips job

Step 2: Click Create → Job, then set the Job name to dev_yourname_fct_trips (always prefix with your schema / name so the shared Jobs list stays readable).

Step 3: Add a task and configure the form fields:

Task name: dbt_fct_trips
Type: Choose dbt.
Source: Select Git provider, then click Edit (or fill in the modal dialog):
Git repository URL: https://github.com/lassebenni/nyc-taxi-dbt-reference.git
Git provider: GitHub
Git reference: Select Branch and enter week-13-ch-4-dbt-solution
dbt commands: Set line 1 to dbt deps (installs project package dependencies) and line 2 to dbt build --select fct_trips (remove any default dbt seed or dbt run lines).
SQL warehouse: hyf-dbt-warehouse (same warehouse as dbt on Databricks).
Warehouse catalog: hyf.
Warehouse schema: dev_yourname (your personal schema).
Environment: Under Dependent libraries / Environment, ensure dbt-databricks is pinned to version 1.8.0.

Create Job Tasks tab: set Source to Git provider pointing to nyc-taxi-dbt-reference and configure dbt task

<aside> 💡 Default commands. Databricks may pre-fill dbt seed and dbt run. Replace them with dbt deps and dbt build --select fct_trips as listed above - this chapter has no seeds, and build runs models plus tests in one step.

</aside>

Step 4: Click Create task, then click Run now.

Databricks Job Runs view displaying green Succeeded run for dev_lasse_fct_trips

Wait for the run to finish. Open the run → dbt task → Output tab. You should see Completed successfully and a Done. PASS=… summary at the bottom:

15:47:41  Running with dbt=1.8.8
15:47:42  Installing dbt-labs/dbt_utils
15:47:42  Installed from version 1.4.1
15:47:42  Up to date!

+ dbt build --select fct_trips
15:47:44  Running with dbt=1.8.8
15:47:46  Registered adapter: databricks=1.8.0
15:47:49  Found 3 models, 13 data tests, 2 sources, 702 macros
15:48:00  Concurrency: 8 threads (target='databricks_cluster')
15:48:00  
15:48:00  1 of 4 START sql incremental model dev_yourname.fct_trips ......................... [RUN]
15:50:10  1 of 4 OK created sql incremental model dev_yourname.fct_trips .................... [OK in 130.72s]
15:50:10  2 of 4 START test not_null_fct_trips_pickup_datetime ........................... [RUN]
15:50:10  3 of 4 START test not_null_fct_trips_trip_id ................................... [RUN]
15:50:10  4 of 4 START test unique_fct_trips_trip_id ..................................... [RUN]
15:50:11  2 of 4 PASS not_null_fct_trips_pickup_datetime ................................. [PASS in 1.03s]
15:50:11  3 of 4 PASS not_null_fct_trips_trip_id ......................................... [PASS in 1.03s]
15:50:22  4 of 4 PASS unique_fct_trips_trip_id ........................................... [PASS in 11.41s]
15:50:22  
15:50:22  Finished running 1 incremental model, 3 data tests in 0 hours 2 minutes and 33.15 seconds.
15:50:22  Completed successfully
15:50:22  Done. PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4

You can also verify that the Job committed a write operation to Delta Lake by running DESCRIBE HISTORY in the SQL Editor:

describe history hyf.dev_yourname.fct_trips;

Look for a row where notebook or job initiated a STREAMING UPDATE or MERGE operation.

With a green run log confirmed, your dbt transformations are now executing inside a Databricks Job task.

<aside> 💡 Recap: You created dev_yourname_fct_trips, configured a dbt task with Git provider source targeting hyf-dbt-warehouse (catalog hyf, schema dev_yourname), triggered a manual run via Run now, and confirmed a green success log.

</aside>

If the run is red, read the task log before you change random fields.

<aside> 💡 Using AI to help: If the Job fails, paste the task error log into an LLM and ask which field is wrong: warehouse, schema, Git branch, or command list (⚠️ Ensure no tokens, connection strings, or PII are included!).

</aside>

⌨️ Hands on: add a schedule (then pause it)

A Job without a Job schedule (a cron or UI trigger) only runs when you click Run now. Production adds a schedule (cron or the UI picker): for example every weekday at 06:00 UTC.

On the shared class subscription, an always-on daily schedule across dozens of students burns warehouse time for no learning gain.

Step 1: On your Job details page, locate the Schedules & Triggers section in the right sidebar and click Add trigger.

Databricks Job details right sidebar showing Schedules & Triggers section and Add trigger button

Step 2: In the Schedules & Triggers dialog, set Trigger type to Scheduled, configure a simple schedule (for example Every 1 Day), and toggle Trigger Status to Paused.

Add Schedule & Trigger dialog: set a periodic trigger and toggle Trigger Status to Paused

Step 3: Click Save. Verify that the Schedules & Triggers section on your Job page now displays the schedule safely saved as Paused - Every day.

Databricks Job details page showing the schedule safely saved in Paused status

<aside> ⚠️ Cost habit. Same rule as clusters: idle scheduled Jobs on a shared bill are real spend. Manual Run now is enough to prove the chapter. The Assignment Extra asks for schedule evidence with the Job paused or teacher-approved.

</aside>

Pausing the schedule ensures your test job does not continue running repeatedly in the background.

<aside> 💡 Recap: You attached a schedule, verified the settings, and immediately paused the Job, proving schedule competency while protecting shared class compute credits.

</aside>

Jobs vs Airflow (keep both in your head)

Question	Databricks Job	Airflow (Week 12)
Where does it run?	Inside the Databricks workspace	Separate orchestrator (local Astro or shared VM)
Best when?	Tasks are Databricks-native (dbt, notebooks, SQL)	Pipeline spans many systems
How you see history	Job runs UI	DAG run / task logs
dbt connection	Task → SQL warehouse	Operator / `uvx` + profile against a warehouse or Postgres

Interview phrasing: "I schedule Databricks-only dbt with a Job dbt task on the SQL warehouse; I reach for Airflow when ingest and transforms live in different systems."

<aside> 🤓 Curious Geek: one Job, many tasks

</aside>

Scheduling dbt Jobs

Why schedule here (on the platform)

Jobs, tasks, and the dbt task type

⌨️ Hands on: create the dbt Job

⌨️ Hands on: add a schedule (then pause it)

Jobs vs Airflow (keep both in your head)

Knowledge Check