Week 13 - Big Data on Databricks
Key terms from Week 13 Chapters 1–5, grouped by the chapter that first introduces each one. Each content chapter links to its entries on first use; use this page when you need a quick definition.
A pattern that keeps one copy of data in cheap object storage and adds a table layer on top, giving warehouse guarantees (transactions, schema, SQL) without maintaining a separate warehouse copy. (Used in The lakehouse idea)
Cheap object storage (on Azure, ADLS Gen2) holding raw files at scale, without built-in tables, transactions, or schema guarantees. (Used in The lakehouse idea)
Azure Data Lake Storage Gen2: scalable object storage on Azure that holds raw files (such as Parquet) at scale for a data lake or lakehouse. (Used in The lakehouse idea)
An open-source columnar file format tuned for analytics queries. Delta Lake stores its underlying data as Parquet files with an added transaction log. (Used in The lakehouse idea)
A database tuned for analytics (Synapse, Snowflake, BigQuery): real tables, transactions, and fast SQL, usually on a separate copy of the data from the lake. The lakehouse tries to give warehouse guarantees without that second copy. (Used in The lakehouse idea)
The table layer Databricks uses on top of Parquet files: a transaction log turns a folder of files into a real table with ACID writes, schema enforcement, time travel, and atomic MERGE. (Used in The lakehouse idea)
An ordered record of every change to a Delta table. Enables ACID transactions, schema enforcement, time travel, and atomic merges. (Used in The lakehouse idea)
All-or-nothing writes so a change fully succeeds or does not happen: readers never see a half-written table, and concurrent writers cannot corrupt each other. On Databricks, Delta's transaction log provides these guarantees on lake files. For how the four letters (Atomic, Consistent, Isolated, Durable) differ across Postgres, cloud warehouses, plain lakes, and Delta, see the optional Lakehouse deep dive. (Used in The lakehouse idea)
Delta rejects writes that do not match the table's column names and types, instead of silently accepting bad data. (Used in The lakehouse idea)
Querying an earlier version of a Delta table (for example "as of yesterday" or before a bad load) using the versions recorded in the transaction log. (Used in The lakehouse idea)
A single all-or-nothing operation that updates matching rows and inserts new ones (MERGE). The primitive incremental dbt models use on Delta. (Used in The lakehouse idea)
Spreading data and work across many machines that cooperate on one job. Worth its overhead only when data genuinely does not fit on one machine or is genuinely too slow there. (Used in The lakehouse idea)
A Databricks cluster setting that shuts the cluster down after a period of idle time, so an unused cluster stops billing. (Used in The lakehouse idea)
A budget notification on the shared workspace that warns when Databricks spend crosses a set threshold. (Used in The lakehouse idea)
The Databricks web app for the class: sidebar areas for notebooks, Catalog Explorer, Compute, SQL, and Workflows (Jobs). One multifaceted platform, not five separate products. You sign in with your Azure account. (Used in Workspace & Unity Catalog)
The left-sidebar area for notebooks and folders (your files in Databricks). Distinct from the Databricks workspace as a whole (the entire web app). (Used in Workspace & Unity Catalog)
The left-sidebar icon that opens Catalog Explorer so you can browse tables. Not the same as the Unity Catalog top-level name (hyf). (Used in Workspace & Unity Catalog)
The left-sidebar page where you start and stop clusters (and see warehouses). Distinct from compute as the machines themselves: this menu is where you control them. (Used in Workspace & Unity Catalog)
The left-sidebar section with the SQL editor and the SQL Warehouses list. dbt connects to a warehouse from here. (Used in Workspace & Unity Catalog)
The left-sidebar section for Jobs: schedule notebooks, SQL, and dbt tasks so they run without you clicking Run. Same workspace as Catalog and Compute. (Named in Workspace & Unity Catalog; practiced in Scheduling dbt Jobs)
An interactive document of code cells (in Databricks under Workspace). You run one cell at a time; output appears under that cell; variables stay in memory for later cells (like a long-lived Python session). On Databricks a cell can be Python (including PySpark) or SQL. The pattern comes from Jupyter Notebooks (.ipynb files); Databricks notebooks use the same cell model in the browser. It does nothing until attached to a running cluster. (Named in Workspace & Unity Catalog; taught in PySpark in Databricks)
Databricks' shared catalog.schema.table namespace across workspaces, clusters, and warehouses (so a notebook and dbt see the same table). Access grants, lineage, and tags are part of the same product, but optional for Week 13. (Used in Workspace & Unity Catalog)
The top level of Unity Catalog's three-part name (catalog.schema.table). For HYF class data the shared catalog is hyf. (Used in Workspace & Unity Catalog)
The middle level of catalog.schema.table: a group of related tables, like a folder. Example: nyc_yellow. (Used in Workspace & Unity Catalog)
The Databricks UI for browsing catalog → schema → table and inspecting columns without running a query. Opened from the Catalog icon in the sidebar. (Used in Workspace & Unity Catalog)
Managed compute for SQL and dbt (SQL editor, dbt builds). Ready in seconds, scales to zero when idle. A notebook attached to it can run SQL cells, but not Python / PySpark. Class warehouse: hyf-dbt-warehouse. (Used in Workspace & Unity Catalog; SQL-in-notebook path in PySpark in Databricks; dbt execution in dbt on Databricks)
A Unity Catalog table whose storage Databricks owns. Dropping it deletes the underlying files. Class tables are managed. (Used in Workspace & Unity Catalog; depth in Going Further)
Table metadata pointing at files in storage you control. Dropping the table leaves the files behind. (Used in Going Further)
Read-Eval-Print Loop: an interactive Python prompt where you type a bit of code, run it, see the result, and keep going without restarting Python. Variables stay in memory between lines. A notebook cell works the same way; a .py script does not (it runs once and exits). (Used in PySpark in Databricks)
The machines that run your work in Databricks: a cluster for Python / PySpark notebooks, or a serverless SQL warehouse for SQL (editor, dbt, and SQL notebook cells). Nothing runs until compute is available and attached. (Used in PySpark in Databricks)
Classic Databricks compute for notebooks and PySpark: one driver coordinates and executors do parallel work. Bills per minute while running. (Used in PySpark in Databricks)
The single machine in a Spark cluster that coordinates work: it plans the job and collects small results. Pulling a huge dataset to the driver with collect() is what causes out-of-memory errors. (Used in PySpark in Databricks)
A worker machine in a Spark cluster that runs part of a distributed job in parallel. (Used in PySpark in Databricks)
The open-source distributed engine that runs data jobs across many machines (a driver plus executors). Databricks runs Spark for you on a cluster; you rarely call Spark directly. You talk to it from Python with PySpark. (Used in PySpark in Databricks)
The entry point to Spark from Python: the object you call as spark.read, spark.sql, and so on. In a Databricks notebook attached to a cluster, Databricks already creates it and names it spark. You do not import or build it yourself (unlike a local PySpark script that uses SparkSession.builder). (Used in PySpark in Databricks)
The Python API for Apache Spark. DataFrames with filter, groupBy, and join, executed across a cluster instead of in one process. You write PySpark in a notebook; Spark is the engine underneath. (Used in PySpark in Databricks)
PySpark builds a plan from transformations (filter, join, groupBy) and only runs it when an action (show(), collect(), count(), write) needs a result. (Used in PySpark in Databricks)
Spark's query optimizer: it sees the full lazy plan and can reorder steps, drop unused columns, and combine work before any data moves across the cluster. (Used in PySpark in Databricks)
A PySpark step that describes work without running it yet, for example filter, select, join, groupBy, or agg. Spark records it in a plan until an action runs. (Used in PySpark in Databricks)
A PySpark operation that triggers execution of the lazy plan, for example show(), collect(), count(), or a write. (Used in PySpark in Databricks)
A distributed table-like dataset in PySpark: named columns and typed rows, spread across executors. Same idea as a pandas DataFrame from Week 4, but the heavy work stays on the cluster. (Used in PySpark in Databricks)
Keep rows that match a condition, for example trips.filter(F.col("payment_type") == 2). A transformation: it builds a plan, it does not run until an action. (Used in PySpark in Databricks)
Keep or rename columns on a DataFrame, for example df.select("pickup_datetime", "total_amount"). A transformation. (Used in PySpark in Databricks)
Combine two DataFrames on a key, for example trips to zones on location id. A transformation. (Used in PySpark in Databricks)
Split rows into groups by one or more column values before an aggregation. A transformation; usually followed by agg. (Used in PySpark in Databricks)
Compute summaries per group after groupBy, for example F.count("*").alias("trip_count"). A transformation until an action runs. (Used in PySpark in Databricks)
An action that prints a small sample of rows in the notebook (show() or show(5)). Safe on large tables because it only samples. (Used in PySpark in Databricks)