Week 13 - Big Data on Databricks
This page is optional. Nothing here is required for Week 13's learning goals or the assignment. Back to the Going Further overview.
Scheduling dbt Jobs covers the required path: one namespaced Job, one dbt task on hyf-dbt-warehouse, one successful manual run, and a schedule you pause on the shared bill. This section is the production-shaped depth around that skill.
Chapter 5 uses Git provider pointing directly to the class reference repository (https://github.com/lassebenni/nyc-taxi-dbt-reference.git, branch week-13-ch-4-dbt-solution). This matches production best practices: the Job runs whatever that commit contains - the same "deploy by merging" idea as Week 12's shared Airflow pull. Workspace folders remain an alternative when working locally inside Databricks without a Git remote. You still set catalog, schema, and warehouse on the task; you still never put a personal token in the repo.
A Job can be a small DAG inside Databricks: for example a notebook task that writes a staging Delta table, then a dbt task that builds marts, with an explicit dependency (task B runs after task A succeeds). That is the Jobs equivalent of Airflow >>. Keep tasks small and idempotent so a retry does not double-write.
flowchart LR
nb["Notebook task<br/>PySpark prep"]
dbt["dbt task<br/>fct_trips build"]
nb --> dbt
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class nb,dbt box;
If you migrated exploration out of interactive cells (see Migrating notebooks to applications), attach the notebook as a notebook task, pick serverless jobs compute or a jobs cluster, and schedule it the same way. Do not leave notebook Jobs attached to a long-lived all-purpose cluster "for convenience" - that is how idle spend appears.
| Situation | Prefer |
|---|---|
| Only Databricks tasks (dbt, notebooks, SQL) | Databricks Job |
| Ingest in blob/ADF + transform in Databricks + notify elsewhere | Airflow (or similar) coordinating the hops |
| Team already runs everything through Airflow | Keep Airflow; call Databricks as one operator/task |
| Learning the lakehouse schedule this week | Job dbt task (required Ch5); Airflow already covered in Week 12 |
Production Jobs should tell someone when they fail. In the Job's settings, open Notifications (or Job notifications) and add your email for On failure (and optionally On success / duration warnings). You do not need a separate monitoring product for a class Job: one address on the Task 3 Job is enough.
<aside>
📝 Optional assignment bonus (Easy): Assignment Task 4 asks for a notification screenshot in task-4/screenshots/job_notification.png plus a one-line note in task-4/BONUSES.md.
</aside>
In Scheduling dbt Jobs you configured a Job manually in the Databricks UI. In enterprise data engineering teams, workflows and projects are defined as code using Databricks Asset Bundles (DABs).
A DAB is a databricks.yml file stored in your Git repository that defines your jobs, tasks, permissions, and target deployment environments (dev, staging, prod):
# databricks.yml snippet
bundle:
name: nyc-taxi-pipeline
targets:
dev:
workspace:
host: https://adb-123456789.azuredatabricks.net
prod:
workspace:
host: https://adb-987654321.azuredatabricks.net
resources:
jobs:
fct_trips_job:
name: "fct_trips_daily_${bundle.target}"
tasks:
- task_key: dbt_build
dbt_task:
commands: ["dbt deps", "dbt build"]
Running databricks bundle deploy -t dev from your terminal or CI/CD pipeline (such as GitHub Actions) validates and provisions the Job automatically.
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 https://hackyourfuture.net/
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.