Workspace & Unity Catalog

In the last chapter you built the mental model: a lakehouse is cheap storage plus a table layer, and Databricks runs on it. This chapter gets you into the Databricks workspace (the web app where you browse data and run jobs). You will learn where tables live, how Databricks names them, and how to confirm the shared data with SQL.

One UI word to recognise now: a notebook (an interactive document of code cells under Workspace) is what you create and run in the next chapter. This chapter stays on Catalog and SQL: find the data and confirm it with one query.

This chapter has three jobs:

Find where tables live in the UI: Catalog Explorer (browse catalogs, schemas, and tables)
Read a three-level table name (catalog.schema.table)
Confirm the shared data with a SQL count(*) on the class warehouse

You already know three-part table names from Postgres (database.schema.table). Databricks calls its shared version Unity Catalog (one shared catalog.schema.table list that every tool in the workspace uses). "Across the platform" means a notebook and dbt both resolve hyf.nyc_yellow.raw_trips to the same table. You do not get a different copy per tool.

By the end of this chapter, you should be able to:

Log into the shared Databricks workspace and find Catalog Explorer
Read a three-level table name (catalog.schema.table) and find that table in Catalog Explorer
Run a SQL count(*) on hyf.nyc_yellow.raw_trips using the shared warehouse hyf-dbt-warehouse

Getting into the workspace

You reach Databricks through the same Azure login you have used since Week 5. Your teacher adds you to the shared workspace. Open https://adb-7405619530719547.7.azuredatabricks.net and sign in with your Azure account.

The workspace is a web app with a left sidebar. The five areas you use this week:

Workspace (sidebar for notebooks and folders): you create a notebook here in the next chapter.
Catalog (sidebar that opens Catalog Explorer): browse tables before you code against them. Databricks organises that list with Unity Catalog (more below).
Compute (sidebar to start and stop machines): you start a cluster from here in PySpark in Databricks.
SQL (sidebar for the SQL editor and warehouses): you use this for the count below, and again in dbt on Databricks.
Workflows (sidebar for Jobs): schedule notebooks, SQL, and dbt without clicking Run by hand. You open this in Scheduling dbt Jobs.

flowchart TB
    WS["Databricks workspace<br/>one multifaceted platform"]
    WS --> W["Workspace<br/>notebooks + folders"]
    WS --> Cat["Catalog<br/>Catalog Explorer"]
    WS --> Comp["Compute<br/>clusters"]
    WS --> Sql["SQL<br/>editor + warehouses"]
    WS --> Jobs["Workflows<br/>Jobs + schedules"]
    Cat --> UC["Unity Catalog<br/>catalog.schema.table"]
    Comp --> CL["cluster<br/>for notebooks"]
    Sql --> WH["warehouse<br/>for SQL + dbt"]
    Jobs --> J["Job + dbt task<br/>unattended builds"]
    classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
    class WS,W,Cat,Comp,Sql,Jobs,UC,CL,WH,J box;

The Databricks workspace home, with the left sidebar and Suggested tables from samples.bakehouse

The home page may list Suggested tables such as sales_customers under samples.bakehouse. Those are Databricks demo tables. Ignore them: this week you use the class catalog hyf, not samples.

This chapter stays on Catalog and SQL. SQL queries run on a warehouse; the next chapter's notebook can use that warehouse for SQL cells, then a cluster for Python. Workflows is the scheduling surface of the same platform: you will use it after dbt works on the CLI. You do not need every other menu item this week.

<aside> ⌨️ Hands on: Open the workspace and click through Workspace, Catalog, Compute, SQL, and Workflows in the left sidebar, so you know where each surface lives. Do not create a Job yet - that is Scheduling dbt Jobs.

</aside>

How Databricks names tables

Every table you touch this week has a three-part name: catalog.schema.table. Read it left to right like a path.

The catalog (top-level container) for class data is hyf.
The schema (a folder of related tables) is nyc_yellow.
The table is the dataset itself: raw_trips.

Written out, that is hyf.nyc_yellow.raw_trips:

flowchart LR
    C["catalog<br/>hyf"] --> S["schema<br/>nyc_yellow"] --> T["table<br/>raw_trips"]

<aside> 📘 Reminder: Same three-part shape as on shared Azure Postgres (nyc_taxi.raw_trips in Week 9 and Week 10). Here the top level is a Unity Catalog catalog (hyf) instead of a Postgres database.

</aside>

That shared list is Unity Catalog. One name works from a notebook and from dbt. So hyf.nyc_yellow.raw_trips is not a magic string: it is the same table everywhere on the platform. That is the only Unity Catalog idea you need for the required path this week.

The class tables are managed (Databricks owns the storage; you do not pick an ADLS path). Managed vs external tables (metadata pointing at files in storage you control) is optional depth on Going Further.

For who tracked tables before Unity Catalog (the "Hive metastore" era), see the optional History of the Lakehouse.

Finding data in Catalog Explorer

The fastest way to see what exists is the Catalog Explorer (the Catalog icon in the sidebar). Expand a catalog to see its schemas. Expand a schema to see its tables. Click a table to see its columns without running a query.

<aside> ⌨️ Hands on: Expand hyf → nyc_yellow → raw_trips in Catalog Explorer and look at its columns.

</aside>

Catalog Explorer expanded to hyf, nyc_yellow, and the raw_trips table with its columns

The columns are the same taxi fields you used on Postgres in Weeks 9–10; only the platform and the row count changed. Also expand hyf.nyc_yellow.raw_zones once (the zone lookup). Under hyf you should also see your personal schema (often empty). Shared nyc_yellow is read-only; you write into your own schema later, in PySpark in Databricks and dbt on Databricks.

Confirm with a SQL count

Prove the data is really there with SQL on the shared warehouse. From the raw_trips table page, open Create → Query. That opens a SQL editor scoped to the table.

The Create menu on a table page, with Query selected to open the SQL editor

Before any query runs, attach the shared serverless SQL warehouse (managed SQL compute that starts in seconds and scales to zero when idle): hyf-dbt-warehouse. If the UI lists another warehouse (for example a Databricks default starter), ignore it.

Attach dialog selecting hyf-dbt-warehouse (2X-Small, Serverless) before running the query

A quick peek confirms columns and that the warehouse is connected. You can run something like SELECT * FROM hyf.nyc_yellow.raw_trips LIMIT 100 and see a small sample:

SQL editor after attaching hyf-dbt-warehouse, showing 100 sample rows from raw_trips

The scale check is the count. Replace the SQL with:

SELECT count(*) FROM hyf.nyc_yellow.raw_trips

<aside> ⌨️ Hands on: From the raw_trips table page, choose Create → Query, attach hyf-dbt-warehouse, and run SELECT count(*) FROM hyf.nyc_yellow.raw_trips. You should see roughly 128 million rows.

</aside>

You should get roughly 128 million rows: the yellow-taxi data for 2023 to 2025 that the rest of the week builds on.

SQL editor running count(*) on raw_trips, returning 128,202,548 rows on hyf-dbt-warehouse

SQL uses this warehouse. In the next chapter you will also attach a notebook to it for a SQL cell, then switch to a cluster for Python / PySpark. You start that cluster when you need Python; dbt later reuses the same hyf-dbt-warehouse connection.

What you can and cannot do

As a student on the shared workspace you can:

Read tables under hyf.nyc_yellow
Run SQL against the shared serverless warehouse hyf-dbt-warehouse
Start and use the shared class cluster hyf-class-cluster (next chapter)
Write tables into your own schema later (notebooks and dbt), not into nyc_yellow

You cannot (and should not try to):

Drop or overwrite tables in nyc_yellow
Create new catalogs, cluster policies, or warehouses
Change other students' schemas

Unity Catalog can also control grants, lineage, and PII tags. Those skills matter in production, but they are optional depth for this week.

<aside> ⚠️ Out of scope: Managed vs external tables, access grants, lineage, and PII tagging live on the optional Going Further page, not the required path.

</aside>

Knowledge Check

https://lasse.be/simple-hyf-teach-widget/mcq.html?bank=week_13_ch2_workspace_quiz&embed=1

If the three-level table name or Catalog Explorer still feels fuzzy, this short demo walks through the workspace UI.

https://www.youtube.com/watch?v=ORMH3pQG8yM

When a query fails with an unfamiliar Databricks error, an LLM can help you decode the message.

<aside> 💡 Using AI to help: Paste the error text (⚠️ Ensure no PII, tokens, or connection strings are included!) into an LLM and ask what it means and what usually causes it. Databricks error text is dense, and this is a fast way to decode it.

</aside>

1. In hyf.nyc_yellow.raw_trips, what is each part?
1. Why does the same table name work from both a notebook and dbt?
1. Under hyf you see nyc_yellow and your personal schema. Which is read-only shared source data?
1. Which warehouse do you pick for the SQL count, and why ignore other warehouses in the list?
1. SQL uses a warehouse. What will notebooks need in the next chapter?

Ready for the next chapter when

The next chapter runs SQL in a notebook on the warehouse, then starts a cluster for Python and reads a Delta table with PySpark. You are ready when:

[ ] You can read hyf.nyc_yellow.raw_trips and name the catalog, schema, and table
[ ] You found raw_trips (and raw_zones) in Catalog Explorer, and saw that nyc_yellow is the shared source schema
[ ] You ran count(*) on raw_trips with hyf-dbt-warehouse (~128M)
[ ] You can say: SQL → warehouse; Python / PySpark → cluster

Extra reading

Navigate the workspace (Databricks): a tour of the sidebar, notebooks, and compute.
What is Unity Catalog? (Databricks): the catalog, schema, and table model in one page (skim the namespace; skip grants for now).

<aside> 📚 For managed vs external tables, Unity Catalog governance (grants, lineage, PII), and other optional depth, see the Going Further page.

</aside>

Next up: PySpark in Databricks, where you start a cluster, attach a notebook, and run your first DataFrame transformations against hyf.nyc_yellow.raw_trips.