When tail -f stops working

On one server, debugging is kamal app logs -f and reading. It works, it is fast, and it stops working at a specific and recognizable moment: when the answer to what happened to that request requires knowing which of six machines handled it.

That moment arrives with the deployment strategies of the last few courses. With three replicas, a worker, a canary and a stable version, one user's session is now spread across processes that each hold a fragment of it.

flowchart TB
  U["One user's request"] --> P["kamal-proxy / ingress"]
  P --> W1["web pod 1 — part of the story"]
  W1 --> J["job enqueued"]
  J --> WK["worker pod — another part"]
  WK --> M["email sent — a third"]
  W1 -.-> Q["Which log has the error?"]
  WK -.-> Q
  M -.-> Q

Log aggregation is the answer to that question: every process ships its logs to one place, where you search across all of them at once.

Three problems it solves

Logs disappear. A container's filesystem dies with the container. A pod that crashed at 2 a.m. and was replaced took its logs with it, and kubectl logs on the pod that replaced it shows a healthy start-up. Shipping logs off the machine as they are written is the only way to have them afterwards.

You cannot search across machines. Nobody greps six hosts by hand properly under pressure. One index over all of them is a different activity.

You cannot correlate. The web request, the job it enqueued and the email it sent are three log streams. Tying them together needs a shared identifier, which is lesson 3.

What this application already does

Read config/environments/production.rb in this repository:

# Log to STDOUT with the current request id as a default log tag.
config.log_tags = [ :request_id ]
config.logger   = ActiveSupport::TaggedLogging.logger(STDOUT)

# Change to "debug" to log everything (including potentially personally-identifiable information!)
config.log_level = ENV.fetch("RAILS_LOG_LEVEL", "info")

# Prevent health checks from clogging up the logs.
config.silence_healthcheck_path = "/up"

These are four decisions, all of them right, and the rest of this course builds on each.

STDOUT instead of a file. A containerized process should log to standard output and let the platform capture it. Writing to a file inside a container means a file nobody collects, on a disk that disappears, that no rotation policy manages. This is the most important line in the block.

log_tags = [ :request_id ] prefixes every line of a request with an identifier shared by all of them. That thread lets you reassemble a story spread over many lines, and lesson 3 extends it.

log_level from the environment. Raising verbosity in production becomes a configuration change instead of a deploy. That matters at 3 a.m., and again on the bill, because debug in production is expensive in volume and dangerous in content.

silence_healthcheck_path drops the /up requests. A probe every five seconds across three pods is fifty thousand lines a day that say nothing. Noise costs storage, money and attention.

What it does not do

The logs are unstructured: human-readable lines meant for a terminal.

[a1b2c3d4] Started GET "/catalog/courses/rails-basics" for 203.0.113.5 at 2026-09-06 10:14:22
[a1b2c3d4] Processing by Catalog::CoursesController#show as HTML
[a1b2c3d4] Completed 200 OK in 47ms (Views: 31.2ms | ActiveRecord: 8.1ms)

That is three lines for one request, in a format a machine must guess at with a regular expression. Ask which requests took longer than a second yesterday and you would have to parse that duration out of prose. Lesson 2 fixes that.

Nothing ships them anywhere either. Right now they go to standard output and, under Kamal, into Docker's log files on the host, where they are subject to rotation and to the host still existing. That is lesson 4.

What this course covers

1. When tail -f stops working (you are here).
2. Structured logs: from prose to data.
3. Correlation: following one request across processes.
4. Shipping: getting logs off the machine.
5. Storing and searching without a large bill.
6. What to log, and what never to log.
7. Putting it together: a logging setup for this app.

Key takeaways

• Aggregation matters when logs outlive their machines, span several of them, or must be correlated across processes.
• This repository already logs to STDOUT with a request-id tag, an env-driven level and silenced health checks, and all four are right.
• Logging to a file inside a container is the common mistake; the platform should capture standard output.
• The gaps are that the lines are prose instead of data, and that nothing ships them anywhere durable.