Week 13 - Big Data on Databricks
This page is optional. Nothing here is required for Week 13's learning goals or the assignment. Back to the Going Further overview.
Background for The lakehouse idea.
Chapter 1 gives you the shape: lake vs warehouse vs lakehouse, a short Postgres ACID table, the eras summary, and what Delta's transaction log adds on top of Parquet. This page goes one level deeper on why those warehouse-style write guarantees matter, letter by letter, across a cloud warehouse, a plain lake, and a Delta table.
ACID is the classic name for four properties a reliable database write should have: Atomic, Consistent, Isolated, and Durable. You already relied on them on the shared Azure Postgres. The lakehouse story is clearer if you see how those same letters hold, or fail, on a cloud warehouse, a plain lake, and a Delta table.
| Letter | Means (taxi-sized) |
|---|---|
| Atomic | A write fully succeeds or does not happen (no half-loaded January 15) |
| Consistent | A commit only succeeds if declared database constraints still hold (not a dbt WHERE) |
| Isolated | Concurrent readers never see a half-done write |
| Durable | After a commit, that result survives a crash or restart |
| Letter | Holds? | Taxi example on Postgres |
|---|---|---|
| Atomic | Yes | Load of 10k Jan 15 trips crashes mid-insert: 0 new rows, not 4k |
| Consistent | Yes | Primary key trip_id cannot be null: an insert without a trip id is refused |
| Isolated | Yes | Dashboard select during the load sees yesterday or full Jan 15, never half |
| Durable | Yes | After commit, restart Postgres: those trips are still on disk |
Cloud warehouses (Snowflake, BigQuery, Fabric, and similar) keep strong Atomic, Isolated, and Durable behaviour on table writes: a failed load should not leave a torn January 15, readers should not see mid-write chaos, and a commit survives. Consistent is often weaker than on Postgres. You can declare a foreign key from trips.pickup_zone_id to zones, but many warehouses treat that FK as informational: zone 99999 may still insert. Teams still paid for a second copy of the data to get those table-write guarantees.
| Letter | Holds? | Taxi example on a cloud warehouse |
|---|---|---|
| Atomic | Yes | Load of Jan 15 fails mid-statement: curated trips does not keep a half-loaded day |
| Consistent | Often no | Declared FK to zones may not reject zone 99999 |
| Isolated | Yes | Dashboard during the load sees previous or complete Jan 15 |
| Durable | Yes | After commit, the warehouse keeps that table version |
A folder of Parquet on ADLS is cheap and scalable. It is not a database. There is no table commit, so ACID does not apply.
| Letter | Holds? | Taxi example on lake files |
|---|---|---|
| Atomic | No | Job dies mid-write of raw/trips/2024-01-15.parquet: a partial file can sit in the folder |
| Consistent | No | No table engine to enforce constraints: a file with missing trip_id can land next to good days |
| Isolated | No | Dashboard opens raw/trips/ during the load: half of Jan 15 is possible |
| Durable | No | Bytes on ADLS are durable, but there is no table commit: a crashed multi-file load can leave an incomplete day |
That gap is why teams copied curated trips into a warehouse.
The lakehouse pitch is not "forget ACID." It is keep warehouse-style Atomic, Isolated, and Durable behaviour (plus schema checks) on the cheap files, without the second copy. On Databricks, the transaction log is what makes that possible. Chapter 1 shows the log in the Delta section; this table is the ACID view of the same idea.
| Letter | Holds? | Taxi example with a Delta table |
|---|---|---|
| Atomic | Yes | Crash mid-load of Jan 15: readers still see yesterday; the bad write never becomes current |
| Consistent | Yes (schema) | Schema enforcement rejects a write that breaks column types (for example "N/A" into fare_amount) |
| Isolated | Yes | Dashboard during the load sees previous version or full Jan 15, never half |
| Durable | Yes | After the transaction log commits version N, that version survives worker failure |
| Era | ACID? | One-line reality |
|---|---|---|
| Database (Postgres) | Yes | Constraints often enforced; commit or roll back on one machine |
| Data warehouse | Mostly | Strong A/I/D on table writes; C often weaker (FKs may be informational) |
| Data lake | No | Files only; half-written days and colliding writers are possible |
| Lakehouse | Yes | Delta transaction log restores warehouse-style A/I/D on lake files |
<aside>
💡 dbt tests are not ACID Consistency. A dbt relationships or not_null test runs as a query after the model builds. It is a valuable safety net. It is not the same as a Postgres primary-key constraint that refuses the insert itself. Our shared taxi tables often skip database foreign keys; do not assume every declared rule is enforced at write time.
</aside>
The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 https://hackyourfuture.net/
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.