Teachers

Logs that outlive the server

Right now, every log line your app writes lives and dies with the container that wrote it. Your hosting platform shows a long stream of logs, but the history is short, a redeploy starts a fresh page, and searching means scrolling. In the first chapter we called the professional answer centralised logging: every log record automatically travels to one searchable home, the moment it is written.

That home will be Grafana Cloud — a hosted observability platform built around open-source tools you will meet constantly in the industry: Grafana for dashboards and search, Loki for log storage. The free tier is genuinely free (no credit card, does not expire) and generous enough for everything in this track: 50 GB of logs per month, kept for 14 days.

<aside> 💭

The other big names in this space — the Elastic/ELK stack, Datadog — work the same way conceptually but have no permanent free tier. The concepts we learn here are the same for other platforms.

</aside>

By the end of this chapter, a log line written by your app — on your laptop or in the cloud — appears in Grafana within seconds, tagged with which environment it came from.

Step 1: Create your Grafana Cloud stack

Sign up at grafana.com — the free tier needs no credit card.
You can skip the automatic setup.
Once logged into the Grafana cloud homepage, Choose your org name on the left and find the Loki instance:

Click on the image to zoom

Open the details page for Loki (look for Grafana Data Source settings. You need three values from this page:
1. The Loki URL — something like https://logs-prod-012.grafana.net
2. The username — a number (your Loki instance ID)
3. An API token — generate one on the same page, with permission to write logs. Copy it immediately - it is shown only once.

<aside> ⚠️

Make sure you generate the API token from the “Sending Logs to Grafana” section as shown in the image above. This will create a token that is able to write logs to the cloud. Double check this when creating a token, it should have "logs:write" scope.

</aside>

<aside> ❗

The token is a secret — the same rule as your database password when you handled sensitive data - it goes into environment variables, never into a file, never into Git. Your repositories are public and bots scan GitHub for leaked tokens within minutes.

</aside>

Step 2: Add the loki4j appender

Your app will push its own log records to Loki over HTTPS. In Logback terms that is just one extra appender — a destination for log records, next to the console. We use loki4j, a small, widely used Logback appender for Loki. Add the dependency to your pom.xml:

<dependency>
    <groupId>com.github.loki4j</groupId>
    <artifactId>loki-logback-appender</artifactId>
    <version>2.0.3</version>
</dependency>

Then create src/main/resources/logback-spring.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <!-- Keep Spring Boot's normal console logging for humans -->
    <include resource="org/springframework/boot/logging/logback/base.xml"/>

    <!-- Send a copy of every log record to Grafana Cloud (Loki) -->
    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>${LOKI_URL}/loki/api/v1/push</url>
            <auth>
                <username>${LOKI_USERNAME}</username>
                <password>${LOKI_API_TOKEN}</password>
            </auth>
            <requestTimeoutMs>15000</requestTimeoutMs>
        </http>
        <batch>
            <maxBytes>65536</maxBytes>
        </batch>
        <labels>
            app = my-app
            env = ${ENVIRONMENT:-dev}
            level = %level
        </labels>
    </appender>

    <root level="INFO">
        <appender-ref ref="LOKI"/>
    </root>
</configuration>

Reading it top to bottom:

The include keeps everything you already have — the console still shows the familiar human-readable lines.
${LOKI_URL}, ${LOKI_USERNAME}, ${LOKI_API_TOKEN} are filled from environment variables at startup. The XML in your repository contains no secrets.
Labels are named fields attached to every record — structured logging in action. app identifies your application (change my-app to your project's name), level carries the log level as data, and env reads the ENVIRONMENT variable, falling back to dev when it is not set. That fallback is why one file serves both your laptop and your deployed service.
The extra <root> adds LOKI as a second destination: every record now goes to the console and to the cloud.
Going deeper (optional): do the logs need to be JSON?

<aside> 💭

Notice what we did not change: your code. log.info(...) stays exactly as it was — where logs go is configuration, not code. Teams redirect logs to new tools without touching a single class.

</aside>

Step 3: Ship from your laptop first

Before involving your deployment, prove the pipe works locally — the feedback loop is seconds instead of a deploy. Set the three environment variables in your IDE run configuration and start the app:

LOKI_URL       = <https://logs-prod-012.grafana.net>    (yours may differ)
LOKI_USERNAME  = 123456
LOKI_API_TOKEN = glc_eyJvIjo...

Checking logs in Loki

Launch your Grafana cloud instance from the Stack homepage
In Grafana cloud, open the menu (top-left button) → Connections → Data sources
Search for the Loki logs data source and click on Explore (compass icon)

In the label filter pick app = my-app → run the query for the last 15 minutes. Hit a few endpoints of your app with Postman to generate lines.

Example log output in Loki

<aside> 🎉

There they are — log lines written on your laptop, searchable in the cloud, tagged env=dev. Your logs just became data.

</aside>

⚠️ Troubleshooting: Nothing arriving

Step 4: Ship from production

Now the same trick where it matters. In your hosting platform's environment variables — exactly where your database credentials already live — add:

LOKI_URL       = (same as dev)
LOKI_USERNAME  = (same as dev)
LOKI_API_TOKEN = (same as dev)
ENVIRONMENT    = production

Commit pom.xml and logback-spring.xml, push, and let last week's pipeline do its job: tests, image, deploy. Once the service is live, hit your public URL a few times, then filter in Explore on env = production.

One app, one config file, two environments — distinguishable with a single click. When something breaks in production at the weekend, the evidence is now waiting for you, searchable, no matter how often the container restarted.

What does this cost?

Nothing — within limits, and the limits teach a real lesson. The free tier ingests 50 GB of logs per month and keeps them for 14 days. Sounds infinite for a training app, and it is — unless you ship noise. This is why the root level in logback-spring.xml says INFO: a chatty DEBUG stream from every framework class is exactly how real companies turn logging into a huge monthly bill. Retention works the same way: storing everything forever is expensive, so every company picks a window; Grafana's free tier picks 14 days for you.

Extra resources

loki4j documentation: the appender's full configuration reference, including the Grafana Cloud example this chapter is based on.
Grafana Loki documentation: how Loki stores and indexes logs — background reading, not needed for this week.
The Twelve-Factor App — Logs: the classic principle that an app should treat logs as a stream and leave storage to the platform — exactly what this chapter puts into practice.

The HackYourFuture curriculum is licensed under CC BY-NC-SA 4.0 *https://hackyourfuture.net/*

CC BY-NC-SA 4.0 Icons

Built with ❤️ by the HackYourFuture community · Thank you, contributors

Found a mistake or have a suggestion? Let us know in the feedback form.

Week 13

Logs that outlive the server

Step 1: Create your Grafana Cloud stack

Step 2: Add the loki4j appender

Step 3: Ship from your laptop first

Checking logs in Loki

Step 4: Ship from production

What does this cost?

Extra resources