Week 13

Observability introduction

Logging in Spring Boot

Structured Logging

Shipping Logs to the Cloud

Debugging with Logs

Health & Metrics

Appendix: Alerting (optional)

Practice

Assignment

• Teachers

Backend Track

When the dashboard watches you back

<aside> 💭

This chapter is optional. Do it after the assignment if you have time and curiosity — it is not required for submission, but it is a satisfying capstone: your system will email you when it is in trouble.

</aside>

Your dashboard from Health & Metrics has one flaw: it only helps while someone is looking at it. Nobody stares at dashboards on a Saturday. An alert flips the direction — instead of you checking the system, the system contacts you when a condition you defined becomes true. This is the mechanism behind every on-call rotation you will ever join.

Build one alert

You already know how to measure trouble — in the dashboard you counted log lines by level. An alert is the same idea, narrowed to one number. Grafana's UI evolves, but creating an alert always comes down to the same four decisions:

1. The query — what to measure. In Alerting → Alert rules → New alert rule, build a Loki query the same way you did for the dashboard: filter app = my-app, add level = ERROR, then Operations → Aggregations → Count. That gives a single number: how many errors in the window.
2. The condition — when is it bad? Threshold: the value is above 0. Any error at all fires the alert — crude, but honest for a service that should have none.
3. The patience — how long must it stay bad? Grafana asks for a pending period (e.g. 2 minutes) so a single blip does not page anyone at 03:00. Set the rule to evaluate every minute.
4. The destination — who gets told, and how? Under Contact points, add your email address. On the free tier, email works out of the box.

Test it like you mean it

An alert you have never seen fire is a hope, not a safety net. Trigger it deliberately. First, add a temporary endpoint to your project:

@GetMapping("/api/test-alert")
public String testAlert() {
    log.error("Test alert: this is a drill");
    return "Error logged";
}

Then:

1. Deploy, hit it a few times, and wait out your pending period.
2. Watch the alert move through its states in Grafana — Normal → Pending → Firing — and check your inbox.
3. Stop hitting the endpoint; watch the alert resolve. Then delete the endpoint — drills should not outlive the drill.

<aside> 🎉

An email from your own production system, telling you it is unwell, seconds after it happened — before any user complained. That is the whole point of this week, closing the loop on its own.

</aside>

A warning about crying wolf

The rookie mistake with alerts is making too many. An inbox full of alerts that "are probably fine" trains you to ignore them — the industry calls this alert fatigue, and it is how real outages get missed. The professional rule: every alert must be actionable. If the honest response to an alert is "do nothing", the alert should not exist — that information belongs on a dashboard, not in your inbox. One good alert (errors are happening) beats ten noisy ones.

Extra resources

• Grafana Alerting docs: alert rules, contact points and notification policies in full.
• Being on-call — Google SRE book: what pager duty actually looks like in mature teams. Advanced, worth skimming before your first job.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.