Week 13

Observability introduction

Logging in Spring Boot

Structured Logging

Shipping Logs to the Cloud

Debugging with Logs

Health & Metrics

Appendix: Alerting (optional)

Practice

Assignment

• Teachers

Backend Track

The detective method

Something is wrong in production. You cannot step through the code, but you have something a debugger never gives you: a recorded history of everything the app said. Working through that history is a method, not a talent, and it is the same four moves every time:

1. Start from the symptom. What exactly did the user see, and roughly when? "Checkout gives an error since this afternoon" gives you a time window and a place to look.
2. Narrow the window. In Grafana Explore, set the time range around the symptom and filter to your app. Fewer lines, more signal.
3. Find the loudest evidence. Filter on level = ERROR. In a healthy app, errors are rare — whatever you find now is almost certainly related.
4. Read outward from the error. The error line tells you what happened. The lines just before it — the INFO and WARN story — tell you what the app was doing on the way there. Cause lives upstream of effect.

<aside> 💡

In Explore you need only three controls: the time range picker (top right), the label filters (app, env, level), and line contains for searching text inside messages — an order ID, an endpoint path, a word from the error. That is enough to solve real incidents.

</aside>

Reading a stack trace

Most ERROR lines come with a stack trace — and juniors often scroll past it because it looks like noise. It is the opposite: it is the machine telling you the exact file and line where things went wrong. Here is one, straight from a shipped log:

ERROR c.h.taskapp.TaskService - Failed to load task 42
java.util.NoSuchElementException: No value present
    at java.base/java.util.Optional.get(Optional.java:143)
    at com.hackyourfuture.taskapp.TaskService.getTask(TaskService.java:31)
    at com.hackyourfuture.taskapp.TaskController.getTask(TaskController.java:24)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(...)
    ... 47 more lines of framework code ...

How to read it:

• The first line names the exception and its message. NoSuchElementException: No value present — someone called .get() on an empty Optional. Half the diagnosis is already done.
• Then, the call chain — most recent call first. Scan down and find the first line from your own package: TaskService.java:31. That is your crime scene. Everything above it is library code doing what your line asked; everything below is how the request got there.
• Ignore the framework lines. The dozens of Spring and Tomcat lines are the request's plumbing. Your package name is the needle — and now you know why LoggerFactory.getLogger(TaskService.class) naming matters.
• Look for Caused by:. Exceptions are often wrapped. If a trace contains Caused by: sections, the last one is the original problem — start there.

So: TaskService line 31 calls Optional.get() on a task that does not exist — someone requested task 42, and it is not in the database. You found this in a running production system without touching a debugger.

Let AI help — the right way

A stack trace is a perfect thing to hand to an AI assistant: it is precise, self-contained, and contains no personal data (check before pasting). The assistant is genuinely good at decoding exception types and suggesting likely causes.

But notice what the assistant is missing: it can see the trace, not your production system. It might reply with three plausible causes — "the ID does not exist", "the row was deleted", "a transaction issue". Which one is true here? Only the logs can say. So the professional loop is:

1. Hypothesis: paste the trace (and the surrounding log lines), ask what could cause this.
2. Verify: go back to Grafana and look for evidence. Is there an earlier line showing task 42 being deleted? Did the request come in with a strange ID? Do other task IDs work fine?
3. Only then, fix. A fix based on an unverified guess is how you "fix" the wrong thing and ship a second incident.

<aside> ⚠️

The assistant sounds equally confident when it is right and when it is wrong — you learned this in the core program as hallucination. In debugging, the logs are your ground truth. AI proposes, evidence decides.

</aside>

Common logging mistakes

Now that you read logs under pressure, you will recognise these — and stop writing them:

• The swallowed exception. catch (Exception e) { } — the error happened, and the logs say nothing. The incident becomes unsolvable. If you catch it, log it (or rethrow it) — never neither.
• Log-and-rethrow. Logging the exception and throwing it up to something that logs it again — the same error appears three times and you chase three incidents that are one. Log where you handle, throw where you don't.
• Context-free messages. log.error("Something went wrong") — wrong where, with what input? Event + context, always: IDs, endpoint, the value that misbehaved.
• Logging in a loop. One line per item in a 10,000-item import buries the day's story — and remember, shipped log volume is money. Log the summary: "Import finished: 10000 rows, 3 skipped".
• The missing level discipline. When everything is ERROR, nothing is — step 3 of the detective method dies. Levels are your future filter; spend them honestly.

Practise the method

💬 A user reports: "I got an error saving my task around 14:30." Here is what level = ERROR plus the minutes before show in Grafana. What is the root cause — and which single log line proves it?

14:29:41 INFO  c.h.taskapp.TaskController - POST /api/tasks received
14:29:41 INFO  c.h.taskapp.TaskService    - Creating task with title=Weekly groceries for user 7
14:29:41 INFO  c.h.taskapp.TaskService    - Task 118 created by user 7
14:30:02 INFO  c.h.taskapp.TaskController - POST /api/tasks received
14:30:02 WARN  com.zaxxer.hikari.HikariPool - Connection is not available, request timed out after 30001ms
14:30:02 ERROR c.h.taskapp.TaskService    - Failed to save task for user 7
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection
    at com.hackyourfuture.taskapp.TaskRepository.save(TaskRepository.java:27)
    ...

• Answer

Extra resources

• Log queries (LogQL) — Grafana Loki docs: the query language behind Explore's filters — useful once the visual builder feels limiting.
• Effective debugging — Google SRE workbook: how professionals stay systematic during incidents. Advanced; read after the assignment.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.