Messaging

Teachers

Introduction

The pieces of this week now point in one direction. Communication Patterns showed that asynchronous communication decouples services. Background Processing showed that in-app background work is lost on restart, cannot retry, and duplicates across instances. Both problems have the same missing piece: a safe place outside the application where work can wait. That piece is a message broker.

Five words

A message is a small piece of data describing work to do or something that happened — for example "send welcome email to [email protected]". Around it, you need exactly five words:

Producer — the application that sends the message
Queue — where messages wait, in order, until someone processes them
Consumer — the application that takes messages from the queue and does the work
Broker — the server that hosts the queues and moves the messages (RabbitMQ, in the next chapter)
Acknowledgement — the consumer's confirmation that a message was processed successfully; only then does the broker delete it

flowchart LR
    P["Producer"] -- message --> Q
    subgraph Broker
        Q[["queue"]]
    end
    Q -- message --> C["Consumer"]
    C -. acknowledgement .-> Q
    style P fill:#e8f3ff,stroke:#2f80ed,color:#123
    style C fill:#eaf8ed,stroke:#2f9e44,color:#123

If the consumer crashes before acknowledging, the broker notices and redelivers the message — to the same consumer later, or to another one.

Why a broker fixes last chapter's problems

Problem with in-app background work	With a message broker
Restart or deploy loses running work	Messages live on the broker, not in your app — the app restarts, the queue is still full
No retry on failure	No acknowledgement → the broker redelivers the message
Work duplicated across instances	Consumers compete for messages from one queue — each message is handed to exactly one of them

There is a bonus: buffering. If 1,000 tasks arrive in one minute, the queue absorbs the burst and the consumers work through it at their own pace — nothing falls over, nothing is dropped.

Two patterns: queue vs pub/sub

Brokers support two fundamental patterns, and knowing which is which will serve you in every messaging conversation:

Pattern	Each message goes to…	Typical use
Work queue	Exactly one consumer (they compete)	Jobs: send this email, resize this photo, generate this report
Pub/sub	Every subscriber gets its own copy	Events: "order placed" → the email service, the invoice service and analytics all react

The work queue has a lovely property: scaling consumers is free. Work piling up? Start a second consumer — the broker automatically spreads the messages between them.

💬 Password-reset emails: work queue or pub/sub? And "a user deleted their account", which three different systems must react to?

Answer

<aside> 💡

At work you may never run a broker yourself. The major clouds offer the same concepts as managed services: Amazon SQS, Azure Service Bus and Google Pub/Sub. Learn the concepts once — the products map straight onto them.

</aside>

<aside> 💭

Plenty of teams start without a broker at all, using a database table as a queue. That is a legitimate first step — a real broker earns its keep when volume grows, retries matter, and several consumers compete for work.

</aside>

Summary

A message broker gives asynchronous work a safe home outside your application: producers drop messages into queues, consumers process them and acknowledge, and anything unacknowledged is redelivered. A work queue delivers each message once; pub/sub copies it to every subscriber. Enough theory — in the next chapter you will run a real broker and push messages through it.

Extra resources

Videos

Reading

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

CC BY-NC-SA 4.0 Icons

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.

Week 14