Week 13 - Big Data on Databricks
This page is optional. Nothing here is required for Week 13's learning goals or the assignment. Back to the Going Further overview.
In PySpark in Databricks you loaded a static Parquet dataset with spark.read.parquet(). In production, raw data arrives continuously in cloud storage buckets (like ADLS Gen2 landing zones).
cloudFiles)Auto Loader incrementally ingests new files as they land in storage without needing manual file listing, state tables, or custom tracking logic. It automatically detects new files, infers schema changes, and handles schema drift:
# Auto Loader reading incoming JSON files from ADLS Gen2 as a stream
df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "dbfs:/tmp/dev_yourname/schema_checkpoints")
.load("abfss://[email protected]/raw_trips/")
)
For managed SaaS and database integrations (Salesforce, PostgreSQL, MySQL, Workday), Databricks provides Lakeflow Connect: point-and-click ingestion connectors managed entirely inside Unity Catalog with automatic Change Data Capture (CDC) into Delta Lake tables.
cloudFiles format, schema evolution, and file discovery options.While dbt on Databricks handles SQL transformations, Databricks also offers a native declarative framework called Delta Live Tables (DLT) (part of Lakeflow Pipelines).
Instead of writing imperative Spark code (df.write.saveAsTable()), you declare desired target tables in SQL or Python and let DLT manage DAG orchestration, retries, and cluster scaling.
DLT's key feature is inline Expectations - data quality constraints enforced at runtime:
CREATE OR REFRESH LIVE TABLE stg_trips
(
CONSTRAINT valid_passenger_count EXPECT (passenger_count > 0) ON VIOLATION DROP ROW,
CONSTRAINT valid_pickup_time EXPECT (pickup_datetime IS NOT NULL) ON VIOLATION FAIL UPDATE
)
AS SELECT * FROM LIVE.raw_trips;
DLT tracks invalid records in pipeline UI dashboards without breaking the entire run unless configured (ON VIOLATION FAIL UPDATE).
EXPECT constraints and violation policies.On Databricks job posts and architecture diagrams you will keep seeing medallion architecture: three layers named bronze, silver, and gold. It is not a new modelling skill. It is Databricks' (and many lakehouse teams') vocabulary for the same raw → staging → marts layering you already practised in Data modeling concepts and Introduction to dbt.
| Medallion layer | Job of the layer | Your Week 13 / Week 10 project |
|---|---|---|
| Bronze | Land raw data as-is (or lightly typed). Keep an audit trail; do not fix business logic here. | Unity Catalog sources hyf.nyc_yellow.raw_trips and raw_zones |
| Silver | Clean, rename, cast, filter garbage. One cleaned table per source. No heavy business joins yet. | dbt stg_trips, stg_zones |
| Gold | Business-ready facts and dimensions analysts and dashboards query. Clear grain. | dbt fct_trips (and other marts) |
flowchart LR
bronze["Bronze<br/>raw_trips"]
silver["Silver<br/>stg_trips"]
gold["Gold<br/>fct_trips"]
bronze --> silver --> gold
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class bronze,silver,gold box;
Why the metal names exist. Lakehouse teams often store each layer as Delta tables on object storage (sometimes in separate Unity Catalog schemas such as bronze, silver, gold). The names travel well in interviews and Databricks docs even when the SQL is still ordinary dbt stg_ / fct_ models.
What Week 13 does not ask you to rebuild. dbt on Databricks ports the existing staging and mart project and makes fct_trips incremental. You do not need separate bronze/silver/gold folders or schemas for the required path. If a teammate says "put that in silver," they mean "clean it in staging, not in the dashboard query."
When production teams go further. Larger lakehouses sometimes (a) land bronze with Auto Loader or Lakeflow Connect, (b) refine silver with PySpark or DLT Expectations, and (c) build gold with dbt incremental marts on a SQL warehouse, often as a multi-task Job. That is the same DAG idea as Multi-task Jobs, not a different architecture.
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 https://hackyourfuture.net/
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.