Week 13 - Big Data on Databricks
This page is optional. Nothing here is required for Week 13's learning goals or the assignment. Back to the Going Further overview.
Everything you built this week is batch processing: data at rest, a job on a schedule. That is the right default for almost all data work. Some problems, though, need the data handled as it arrives: fraud detection that cannot wait for tonight's run, a live dashboard, sensor readings that lose value by the hour. That is streaming.
Be honest about scope: a production streaming role is a specialization built over months of operational practice (delivery guarantees, consumer groups, offsets, backpressure). What follows is the shape of streaming on Databricks, not a streaming-engineer track.
The micro-batch mental model. Spark's Structured Streaming treats a stream as a table that never stops growing. It processes new rows in small batches every few seconds and repeats, so your batch DataFrame skills transfer directly. The API mirrors the batch API: spark.read becomes spark.readStream, and .write becomes .writeStream (a continuous query). Here is a minimal demo against a built-in synthetic source:
events = (
spark.readStream
.format("rate") # test source: N rows per second
.option("rowsPerSecond", 5)
.load()
)
query = (
events.writeStream
.format("delta")
.option("checkpointLocation", "dbfs:/tmp/dev_yourname/events_demo_checkpoint")
.toTable("your_schema.events_demo")
)
Use a dbfs: path for the checkpoint (not driver-local /tmp): on a shared cluster, driver /tmp is not durable across restarts. Replace dev_yourname with your assigned schema name.
The transformation in the middle would be ordinary DataFrame code, identical to a batch job. writeStream starts a continuous query that runs until you stop it, and checkpointLocation is how it remembers its progress across restarts.
Stop the query. After you have seen rows land in the Delta table, cancel the streaming cell / stop the query in the notebook, then terminate idle cluster compute. A forgotten stream keeps burning the shared bill.
When to stream. Rarely. Streaming adds permanent complexity (always-on compute, checkpoints, harder debugging), so it pays only when low latency is a genuine hard requirement. If the answer to "how fresh does this need to be?" is measured in hours, it is a batch job.
<aside>
📝 Optional assignment bonus (Harder): Assignment Task 4 asks you to run this rate-source demo, write the results to a Delta table, describe what you observed in task-4/BONUSES.md, and confirm you stopped the query.
</aside>
readStream, writeStream, and checkpointing in context.Kafka is the distributed event-streaming platform most streaming jobs are built on. A one-week introduction teaches vocabulary without the operational judgement the tool actually requires, so it lives here for anyone pursuing the specialization.
Event Hubs is the Azure-native managed streaming service, and it exposes a Kafka-compatible endpoint, so the same client libraries talk to it. On Azure this is usually the broker you reach for rather than running Kafka yourself.
PySpark in Databricks shows the class version of a Spark cluster: one driver and one or more executors. That is enough for the week. Under the hood, Spark always has a third role that Databricks mostly hides.
| Role | Job | What you see in class |
|---|---|---|
| Driver | Plans the job, talks to your notebook, collects small results | The machine your notebook session is tied to |
| Executors | Run tasks in parallel on partitions of the data | The workers that make a groupBy over 128M rows finish |
| Cluster manager | Allocates machines for the driver and executors | On open-source Spark: YARN, Kubernetes, or Spark's own standalone manager. On Databricks: the platform (Compute policies, autoscaling, SKUs) |
When you attach a notebook to the shared class cluster, Databricks has already asked its cluster manager for a driver plus workers. Your PySpark cell does not talk to YARN; it talks to the driver, and the driver schedules tasks on the executors.
flowchart LR
nb["Notebook cell<br/>(PySpark)"] --> drv["Driver"]
cm["Cluster manager<br/>(Databricks Compute)"] --> drv
cm --> ex["Executors"]
drv -->|tasks| ex
ex -->|partial results| drv
classDef box fill:#e8f0fe,stroke:#4285f4,color:#111;
class nb,drv,cm,ex box;
You do not need to configure a cluster manager for Week 13. Know the three roles so "the driver OOMed" and "add more workers" mean something when you read docs or logs later. Creating and sizing clusters yourself is optional depth on Creating your own compute.
Chapter 1 treats a Delta table as "Parquet files plus a transaction log." This interactive walk-through makes that concrete: a mid-write Parquet folder can expose half of January 15; the same write behind a Delta log either commits cleanly or stays invisible to readers.
Click through the steps once. Then use the docs below when you want the formal names (checkpoint, _delta_log, time travel versions).
MERGE on top of Parquet files. This is the substrate under everything you did this week.merge vs insert_overwrite and when a partitioned overwrite beats a merge at very large scale.The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 https://hackyourfuture.net/
Built with ❤️ by the HackYourFuture community · Thank you, contributors
Found a mistake or have a suggestion? Let us know in the feedback form.