Week 13 - Big Data on Databricks

The lakehouse idea

Workspace & Unity Catalog

PySpark in Databricks

dbt on Databricks

Scheduling dbt Jobs

Practice

Assignment

Gotchas & Pitfalls

Slides (PDF)

Career relevance: Week 13

Glossary: Week 13

History of the Lakehouse

• Going Further

History of the Lakehouse

This page is optional. Nothing here is required for Week 13's learning goals or the assignment. It traces how data teams moved from single-server warehouses to distributed lakes, and why the lakehouse idea (cheap files plus a table layer on one platform) is the shape of the stack you use in Databricks this week. Read it in one sitting, or come back when someone at work says "we should just put it in the lake."

Week 9's [History of SQL & Warehouses](../Week 9/week_9__history_of_sql_and_warehouses.md) covers how analytical databases and star schemas evolved. Week 12's History of Data Orchestration covers how pipelines got scheduled and made idempotent. This page covers the storage and compute side: what happened when the data outgrew one machine, and how the industry tried to fix it.

Hadoop and MapReduce

Background for The lakehouse idea

By the mid-2000s, web companies had a new problem. Click logs, ad impressions, and search indexes were too large to load into a single database, yet too valuable to delete. Apache Hadoop, inspired by Google's 2004 MapReduce paper, offered a way out: store files on a cluster of cheap machines and run batch jobs that split the work across them.

The core pattern was MapReduce: a map step processes each chunk of data in parallel, a shuffle moves related records together, and a reduce step combines partial results. Yahoo, Facebook, and LinkedIn built their early data platforms on this model. It worked, but every pipeline was a custom Java program, and a simple join could require three separate MapReduce jobs wired together by hand.

For you as a data engineer today, Hadoop is mostly history, but its assumptions still shape the tools you use. The idea that storage should be cheap and separate from compute, that jobs should be rerunnable on immutable input files, and that scale means adding machines, not buying a bigger server all came out of this era. When you run PySpark on a cluster this week, you are using a direct descendant of that design, with a much better programming model on top.

<aside> 🤓 Curious Geek: Why "Hadoop"?

The name comes from the toy elephant belonging to the son of co-creator Doug Cutting. The elephant mascot stuck so well that "Hadoop" became shorthand for the whole big-data stack, even after MapReduce itself faded.

</aside>

Data lakes on object storage

Background for The lakehouse idea

Hadoop's default storage layer was the Hadoop Distributed File System (HDFS): files replicated across cluster nodes. That worked inside one data centre, but cloud providers offered something cheaper and more durable: object storage (Amazon S3 launched in 2006, Azure Blob and Google Cloud Storage followed). Teams started landing raw files in S3 as a data lake: a cheap place to dump everything, structured or not, and figure out the schema later.

The lake solved cost and flexibility. It also created a new kind of pain. Without a table layer, a "lake" is just a folder of Parquet and JSON files. There is no UPDATE, no enforced schema, no easy way to know which file version is current, and no ACID guarantee if two jobs write at once. Data engineers spent the 2010s building glue: Hive metastores, partition folders named year=2024/month=03/, and nightly compaction jobs to merge small files.

Hive (and later engines such as Presto / Trino) was the bridge that made lakes queryable with SQL before Spark SQL and open table formats matured: register folders as tables in a metastore, then run SQL over files that were never loaded into a warehouse. That SQL-on-files pattern is still what you feel when Catalog Explorer or dbt points at Delta tables on object storage. Hive's metastore was also fragile at scale (one catalog per cluster or workspace), which is the problem Unity Catalog later attacked for Databricks.

If you have ever wondered why Delta Lake talks so much about transactions and time travel, this is the wound it is dressing. The lake was cheap; operating it like a warehouse was not.

When warehouses hit the scale wall

Background for The lakehouse idea

While lakes soaked up raw logs, data warehouses (the columnar analytical databases from Week 9) kept handling curated reporting. Redshift, BigQuery, and Snowflake made it easy to run SQL on billions of rows, as long as you paid for the compute and kept data inside their format.

The split looked clean on a slide: lake for raw, warehouse for curated. In practice, teams maintained two pipelines, two copies of the same data, and two bills. Every new dataset raised the same argument: land it cheap in the lake, or model it properly in the warehouse? ETL jobs multiplied. Governance became a guessing game about which copy was authoritative.

For you as a data engineer today, that split is why job postings still ask for both "data lake" and "warehouse" experience even when the product marketing says lakehouse: many teams are mid-migration, running duplicate pipelines while they consolidate.

The lakehouse pitch, which you meet in The lakehouse idea, is a response to that split: keep cheap object storage, but add warehouse-grade table semantics on top so SQL, dbt, and Spark can all target the same files. You are not learning a third category for fun. You are learning the industry's attempt to undo a decade of duplication. For how ACID holds or fails on a database, warehouse, lake, and lakehouse (with taxi examples), see the optional Lakehouse deep dive.

Spark and the unified engine

Background for PySpark in Databricks

Apache Spark started in 2009 at UC Berkeley's AMPLab as a faster alternative to MapReduce. Where MapReduce forced every step to spill to disk, Spark kept intermediate results in memory when it could, and exposed a richer API: map, filter, join, groupBy, all composable in one program.

Spark also unified batch and streaming behind one engine. The same DataFrame API could process a historical Parquet folder or a live Kafka stream (you will see the streaming angle on Going Further). That mattered because teams were tired of maintaining separate batch and real-time stacks.

Databricks, founded in 2013 by several Spark creators including Matei Zaharia, commercialised managed Spark before most companies wanted to operate their own clusters. The managed notebook, cluster autoscaling, and integrated catalog you use this week are the product layer on top of the open-source engine.

For you as a data engineer, Spark is the default answer when SQL alone cannot express the transformation (nested JSON, heavy Python UDFs, machine-learning features) or when the data is too large for one machine. It is also the reason lazy evaluation exists: Spark builds an execution plan across the cluster before running anything, exactly the behaviour you practised in PySpark in Databricks.

Delta Lake and open table formats

Background for The lakehouse idea and dbt on Databricks

Plain Parquet files in a lake are not enough for warehouse workflows. You need ACID transactions (all-or-nothing writes), schema enforcement, and a way to update and delete rows without rewriting the whole dataset. Delta Lake, open-sourced by Databricks in 2019, adds a transaction log on top of Parquet so reads and writes stay consistent.

Delta is one of several open table formats from the late 2010s. Apache Iceberg (originally Netflix, 2018) and Apache Hudi (Uber, 2017) solve similar problems with different trade-offs. Warehouses and engines increasingly read all three; the format war is still settling, but the direction is clear: files on object storage, with table semantics in a thin metadata layer.

That layer is what makes your incremental dbt model work this week. When is_incremental() is true and dbt issues a MERGE into fct_trips, Delta handles row-level upserts on Parquet files underneath. Without it, you would be back to delete-and-reinsert hacks or full-table rewrites.

<aside> 🤓 Curious Geek: Parquet before Delta

Parquet itself came out of the Hadoop ecosystem (Twitter and Cloudera, 2013) as a columnar file format optimised for analytics scans. Delta did not replace Parquet; it wraps it with a _delta_log folder that records every commit. Your data is still Parquet; the log is what turns a folder into a table.

</aside>

Databricks and the lakehouse name

Background for The lakehouse idea and Workspace & Unity Catalog

Databricks popularised the term lakehouse in a 2020 blog post and paper: the name combines lake (cheap object storage) and warehouse (tables, SQL, transactions). The technical claim underneath is that a good table format over object storage can do the warehouse's job without maintaining a second copy of the data. Warehouse vendors and cloud platforms contested the framing (for example Snowflake's public messaging treated the lake as something warehouses already covered, while AWS/Azure/GCP each pitched managed lake + warehouse stacks). Read those takes knowing who wrote them. The vocabulary stuck because it named a real pain: teams wanted one copy of the data and multiple engines on top.

The Databricks workspace you log into this week is that product story made concrete. Catalog Explorer shows governed tables, SQL warehouses run dbt and ad hoc queries, and clusters run PySpark notebooks, often against the same underlying Delta files. You are not learning a random vendor UI; you are learning the current default shape of a "big enough to matter" analytics stack in many European data teams.

Managed platforms also shifted the ops burden. In the Hadoop era, someone on the team operated HDFS, YARN, and Hive. On Databricks, you still need to understand clusters, costs, and permissions, but you are not SSH-ing into worker nodes to clear a full disk. That trade-off (less ops, more platform lock-in and spend vigilance) is the modern bargain.

Unity Catalog and governed lakes

Background for Workspace & Unity Catalog

Before Unity Catalog, Spark workspaces usually tracked tables in a Hive metastore: one catalog per workspace, with permissions bolted on per cluster. That model did not scale when teams shared many workspaces and needed one name for the same table everywhere.

Cheap storage without governance also recreated the "data swamp" problem: thousands of tables, unclear owners, no consistent permissions. Unity Catalog, generally available in 2022, replaced the per-workspace metastore with an account-level catalog. It brought a three-level namespace (catalog.schema.table), centralised access control, and audit logging across Databricks workspaces.

The hyf catalog you query this week is an instance of that model. Managed tables live under Databricks control; external tables point at storage you own elsewhere (optional depth: Going Further). Either way, the catalog is the map that tells humans and tools where data lives and who may read it. The three-level name is the part you use every day; grants, lineage, and column tags are the rest of that story (optional this week).

For you as a data engineer, Unity Catalog is the answer to "how do we govern a lake without copying everything into a separate warehouse?" It is younger than Postgres schemas or Snowflake databases, but the mental model transfers: name things clearly, grant least privilege, and treat the catalog as the source of truth for what exists.

Putting it together

Background for the whole week

Each row is one generation of the story above: what the technology solved, and the gap it left for the next generation to close.

Extra reading

• MapReduce: Simplified Data Processing on Large Clusters (Google, 2004): the paper that started the distributed batch era.
• Spark: Cluster Computing with Working Sets (Zaharia et al., 2010): the original Spark research paper; short and readable.
• Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores (Armbrust et al., VLDB 2020): how the transaction log works on top of Parquet.
• What is a Data Lakehouse? (Databricks, 2020): the post that named the architecture you practise this week.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.