Introduction to Big Data and Streaming

Apache Spark Core Concepts

Databricks

Streaming Theory

Streaming Platforms: Kafka and Azure Event Hubs

Practice

Assignment

Gotchas & Pitfalls

Week 13 Lesson Plan (Teachers)

Streaming Platforms: Kafka and Azure Event Hubs

Content coming soon...

Suggested Topics

• What is Apache Kafka: a distributed event streaming platform (theory/conceptual)
• Kafka architecture: brokers, topics, partitions, producers, consumers, and consumer groups
• Offsets and delivery guarantees: how Kafka tracks message consumption
• Serialization: encoding messages as JSON (Optional/advanced: Avro serialization)
• Azure Event Hubs: the Azure-native streaming service, Kafka-compatible API
• Azure Event Hubs as a practical example: creating an Event Hub namespace, sending and receiving messages
• When to use Azure Event Hubs vs self-managed Kafka
• Event Hubs Kafka compatibility layer: using Kafka client libraries to talk to Event Hubs
• Error handling in streaming: dealing with bad messages and consumer lag

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0

*https://hackyourfuture.net/*

Found a mistake or have a suggestion? Let us know in the feedback form.