Structured Logging at Scale

Structured logging is what makes logs usable once you have more than one service and one machine. A plain-text log line is a sentence a human reads; at scale, across dozens of services and millions of lines, sentences are ungreppable noise. Structured logs, machine-readable records with consistent named fields, turn logs into something you can filter, aggregate, and join to traces. The catch is that structured logs are still logs, and log volume is the cost that bites back, so the real skill is logging structured, useful, and affordable all at once.

Logs are the observability pillar teams treat most carelessly, and it shows up as either unsearchable text or a shocking bill. Both are avoidable.

Why logging changes at scale

One service on one machine, grep on a text file is fine. Add dozens of services across many machines and containers that come and go, and free-form text collapses: there is no file to grep, no consistent format to match, and no way to follow one request across services. The thing that worked at small scale actively fails at large scale.

Structured logging is the fix, and observability at scale treats logs as one of three pillars alongside metrics and traces. Each answers a different question; logs answer “what exactly happened in this specific event,” and they can only do that at scale if they are structured. This post is part of the Observability series.

What is structured logging?

Structured logging emits each log as a machine-readable record with named fields, almost always JSON, instead of a free-form text sentence. Rather than User 123 failed to check out because payment declined, you emit {"level":"error","service":"checkout","user_id":"123","event":"checkout_failed","reason":"payment_declined","trace_id":"abc"}. The information is the same; the difference is that every field is addressable, so you can filter and aggregate on it directly.

That addressability is the entire value. A text line has to be parsed with fragile regexes that break the moment the wording changes; a structured record is queried by field name, reliably, forever. At scale this is not a nicety, it is the difference between logs you can actually search during an incident and logs that are write-only.

Why is structured logging better than plain text?

Because plain text does not survive scale: you cannot reliably filter, aggregate, or correlate free-form sentences across many services. Structured logs give you consistent named fields, so “show every error for this trace_id” or “count failures by reason” become direct queries rather than regex archaeology. When you have millions of lines from dozens of services, that queryability is what makes logs useful instead of a haystack.

	Plain-text logs	Structured logs
Format	Free-form sentences	Named fields (JSON)
Filtering	Fragile regex	Query by field
Aggregation	Hard/unreliable	Direct (count/group by field)
Correlate across services	Very hard	Join on trace_id
Survives scale	No	Yes

The clinching feature is correlation. Put a trace_id field on every log line and your logs join directly to your traces: from a slow trace you jump to exactly the log lines for that request, across every service it touched. Free-form text cannot do this reliably; structured logging makes it trivial.

What fields should every log line include?

Every log line should carry a consistent core: a timestamp, a level, the service name, and a correlation ID, ideally the trace_id so logs tie back to traces. Beyond that, include the request-scoped context that makes an event meaningful, route, status, relevant IDs as fields. The non-negotiable is consistency: the same field names and shapes across every service, so a single query works everywhere.

This shared schema is the logging equivalent of a shared contract. Decide the core fields once, provide a logging library or wrapper that emits them automatically, and every service speaks the same log language. Without that, each team invents its own fields and you lose the cross-service queryability that was the whole reason to go structured.

How do you control logging costs at scale?

Control the volume, because volume is what you pay for. Log at the appropriate level (never leave debug logging on in production), sample high-volume repetitive log events rather than emitting every one, push high-cardinality detail into fields on fewer lines instead of spamming many lines, and pick a backend whose cost model matches your volume. Logging everything at full verbosity is the standard way observability bills spiral.

The specific levers, in order of impact:

• Right log level in production. Debug and trace logging in prod is often the single biggest source of volume. Default to info and above.
• Sample repetitive events. If the same event fires thousands of times a second, keep a representative sample, not every instance.
• Detail in fields, not extra lines. One structured line with rich fields beats ten narrative lines; and keep genuinely high-cardinality data (per-user, per-request) in fields on sampled lines, not on every line.
• Choose the backend for your volume. Systems like Loki index labels rather than full log content, which changes the cost curve; match the tool to how much you log and how you query.

The mindset is that logs are the highest-volume and often cheapest-per-line but most-voluminous pillar, so discipline pays off fastest here. This is the log-side version of the cardinality discipline from Taming Prometheus Cardinality: the default behavior scales your bill with your traffic unless you bound it deliberately.

When should you use logs versus metrics versus traces?

Use each pillar for what it is built for: metrics for bounded aggregates and alerting, traces for following one request across services, and logs for the detailed record of what happened in a specific event. Reaching for the wrong one is how teams end up with exploded metric cardinality or unsearchable logs. The three are complementary, not interchangeable.

A quick decision guide:

• Metrics answer “how much, how often, is it within bounds” across time, cheaply and at low cardinality. Use them to alert and to see trends. Do not put per-request detail in them.
• Traces answer “where did the time go for this request across services.” Use them to find the slow hop. See OpenTelemetry sampling.
• Logs answer “what exactly happened in this specific event,” including the high-cardinality detail (which user, which request, what values) that metrics cannot hold.

The three connect through the trace_id: a metric alert tells you something is wrong, a trace shows you which service, and the structured logs for that trace_id tell you exactly what happened there. Designing them to interoperate, especially putting the trace_id on every log line, is what turns three separate tools into one coherent debugging workflow. Using logs to do a metric’s job (counting things by grepping) or a metric to do a log’s job (per-entity detail in labels) is where observability goes wrong and gets expensive.

Which log level should a line actually use?

Levels are the oldest idea in logging and the most inconsistently applied. In most codebases they encode how the author felt rather than what a reader should do, which is why “turn up the log level” so rarely helps during an incident.

A definition that makes levels operationally useful ties each one to an action:

Level	Means	Who acts
ERROR	The operation failed and a human should eventually look	On-call, or a ticket
WARN	Something unexpected, but the operation succeeded	Nobody now; reviewed in aggregate
INFO	A significant business event occurred	Nobody; read when investigating
DEBUG	Developer detail for reproducing a problem	The engineer debugging, on demand

Two rules make the scheme hold. ERROR means the operation failed — not “something surprising happened,” not “a retry was needed.” A retry that eventually succeeded is WARN at most, because the system did its job. Logging recoverable conditions as errors is the single fastest way to make error counts meaningless, and once meaningless they cannot be alerted on.

And WARN is the level to be suspicious of. In most codebases it accumulates conditions nobody has ever acted on. A periodic review asking “has anyone ever done anything because of this line?” typically promotes a few to ERROR and demotes the rest to INFO or deletes them.

Two more habits worth adopting. Log the error once, at the boundary where it is handled — not at every layer as it propagates. Logging at each level of the stack produces five lines for one failure, inflating both volume and apparent error counts. And include the cause, not just the symptom: the exception type and the relevant identifiers, so the reader does not have to correlate across services to learn what actually happened.

The pragmatic note on DEBUG in production: rather than a global switch, make verbosity selectable per request — a header or a flag that enables detailed logging for one trace. You get full detail for the case under investigation without paying for it across all traffic, which is the only version of “debug in production” that survives a cost review.

How do you migrate an existing codebase to structured logging?

Greenfield structured logging is easy. The real situation is a codebase with thousands of printf-style log statements accumulated over years, and a big-bang rewrite is neither affordable nor safe.

The approach that works is incremental and prioritised by what you actually query:

1. Add the structured logger alongside the old one. Both write to the same destination. Nothing breaks, and new code can be correct immediately.
2. Make new code structured, enforced in review. This stops the problem growing, which matters more than any migration progress in the first month.
3. Convert by query pain, not by file. Find the log lines you actually grep during incidents — the ones in your runbooks — and convert those first. A small fraction of log statements account for nearly all real usage.
4. Add the context fields centrally, via middleware or a logging context, rather than at each call site. Trace ID, request ID, user ID, and service name should attach automatically; asking every engineer to remember them guarantees inconsistency.
5. Leave the long tail alone. Log lines nobody has ever read do not need converting. They need deleting, and the migration is a good moment to notice how many there are.

Two decisions to make early because they are expensive to change later. Fix the field names and their meanings up front — user_id in one service and userId in another defeats the entire point, since cross-service queries are the main reason to do this. A short schema document, or better a shared logging library that supplies the standard fields, prevents years of inconsistency.

And decide what the message field is for. In structured logging the message should be a stable, low-cardinality string describing the event class — "payment_failed", not "Payment failed for user 12345 after 3 retries". The variable parts belong in fields. This is what makes it possible to aggregate by event and filter by field, which is the capability you migrated for; a message that embeds variables is a plain-text log wearing a JSON envelope.

What must never appear in a log line?

Logs are the highest-risk telemetry you produce, because they are the least structured, the most widely readable, and the most likely to be shipped to a third party. Metrics rarely leak anything; logs leak everything, and they do it durably.

Treat this list as non-negotiable:

• Credentials of any kind — passwords, API keys, tokens, session identifiers, cookies. Note that logging an entire request or response object is the usual way these arrive, not a deliberate decision to log a token.
• Full request and response bodies. Convenient during debugging and a permanent liability in production. If you need them temporarily, gate them behind a flag with an expiry.
• Personal data beyond what you can justify — full names, email addresses, phone numbers, addresses, payment details. Logs are frequently retained longer than your privacy policy promises and are rarely covered by deletion pipelines, which turns a debugging convenience into a compliance problem. The deletion fan-out that has to reach them is described in GDPR Deletion in Event-Driven Systems.
• Anything you would not paste into a shared document. Logs are searched by support, exported to vendors, and pulled into incident channels.

Two mechanisms make this durable rather than aspirational. Redaction at the logging library, so sensitive field names are stripped centrally rather than depending on every engineer remembering at every call site. And a CI check that fails on obviously dangerous patterns — logging a whole request object, or a field named password, token, or secret.

The mindset that prevents most incidents: log identifiers, never values. A user ID rather than an email address. A payment ID rather than a card number. You keep full debuggability, because the identifier lets you look up whatever you need in the system of record, and the log itself stays safe to leak.

How do you control logging cost without losing debuggability?

Log volume grows superlinearly with traffic, and structured logs are larger per line than plain text. The bill arrives before anyone notices the growth, and the reflex — turn the log level down — trades away exactly the detail you need during an incident.

Better levers, roughly in order of value:

Lever	What it does	Cost to debuggability
Sample high-volume INFO	Keep a fraction of routine successful requests	Minimal — successful requests are interchangeable
Never sample errors	Keep 100% of WARN and above	None; this is the rule that makes sampling safe
Drop duplicate context	Stop repeating fields already on the span or the trace	None
Shorten retention, tier storage	Hot for days, cheap object storage for months	Slower access to old data, not loss
Remove debug logs from hot paths	Delete lines nobody has ever read	None if genuinely unread
Lower the global log level	Blunt reduction	High — avoid

The principle underneath: sample by outcome, not by rate. A uniform 10% sample discards 90% of your errors, which is the opposite of what you want. Keeping every error and sampling only routine successes typically cuts volume by most of its bulk while losing nothing that matters during an investigation.

One more rule that saves both money and confusion: do not log what a metric should count. A line emitted per request purely so someone can count requests later is the most expensive counter ever built. Counting is what metrics are for; logs are for the context around a specific event you will want to read.

A structured-logging checklist

For logging that works at scale:

• Logs are emitted as structured records (JSON), not free-form text.
• A shared core schema (timestamp, level, service, trace_id) is used identically across all services.
• Logs carry the trace_id so they correlate with traces.
• Production log level is info-and-above; debug is off unless deliberately enabled.
• High-volume repetitive events are sampled, not logged in full.
• High-cardinality detail lives in fields, not in the metric labels it would break.
• The backend’s cost/index model fits your volume and query patterns.

A last point that ties the levels back to cost: log level and sampling policy should be decided together, not separately. Keeping every ERROR and WARN while sampling INFO heavily is coherent; sampling uniformly across levels quietly discards the errors you most need. Whichever backend you use, express the retention and sampling rules in terms of the levels above, so the cheap-to-drop and never-drop categories are explicit rather than emergent. Written down once, that policy also gives you a defensible answer the next time someone asks why the observability bill grew, which is a conversation every team has eventually.

What I’d do differently

The mistake I have made is treating logging as an afterthought, sprinkling print-style lines during development and shipping them, then discovering at scale that the logs are unsearchable text and the bill is alarming. Logs feel free to add, exactly like metric labels, and exactly like them they are not free at scale.

If I were setting up logging for a new system, I would define the shared structured schema and a logging wrapper on day one, so every service emits consistent, correlated, structured records without each developer deciding the format. And I would set production log levels and sampling deliberately rather than discovering the volume via the invoice. Structured logging is not more work than bad logging; it is the same work done once, correctly, so that when you are debugging an incident at 2 a.m. the logs are a queryable record instead of a wall of text.

Sources

• Grafana Loki, logging at scale and label design: grafana.com/docs/loki/latest/get-started
• OpenTelemetry, Logs: opentelemetry.io/docs/concepts/signals/logs
• Google SRE Book, Monitoring and logging practices: sre.google/sre-book/monitoring-distributed-systems