Jaeger Tracing for Cross-Service Debugging

Jaeger tracing exists to answer one question that logs and metrics cannot: when a request crossed eight services and was slow, which service was slow? It stitches per-service timing data into a single end-to-end trace, so a problem that used to take hours of cross-team log archaeology becomes one glance at a waterfall. The catch nobody mentions up front is sampling, which quietly decides whether the trace you need was ever recorded.

In a monolith, a profiler tells you where time went. In a distributed system, the time is scattered across services, machines, and teams. Distributed tracing reassembles that scattered timing into one picture, and it is close to indispensable once you pass a handful of services.

Why distributed tracing matters

Three signals make up observability: logs (what happened), metrics (how much, how often), and traces (where the time went across services). Most teams have the first two and discover they cannot answer the third when a request is mysteriously slow.

The failure pattern is familiar: a dashboard shows p99 latency spiked, but the latency is spread across a call chain, and each service’s logs only show its own slice. Without tracing, you are correlating timestamps across teams by hand. With it, you open the trace and the slow span is right there. This post is part of the Observability series and pairs with timeout budgets across service chains, which the traces here help you debug.

What problem does Jaeger tracing solve?

Jaeger answers “where did the time go” for a request spanning many services. Every service attaches a shared trace ID and records its own span; Jaeger collects those spans and assembles them into one end-to-end timeline. Instead of guessing which service caused a slow request, you see the exact hop and how long it took.

This is the capability that does not exist without tracing. Logs are per-service and per-line; metrics are aggregate. Neither reconstructs the path of a single request across the system. The trace does, and that reconstruction is what turns a multi-team debugging session into a single lookup.

What is the difference between a trace and a span?

A span is one unit of work within one service, carrying a name, a start time, a duration, and metadata (tags, logs, status). A trace is the tree of spans for a single request as it travels across services. The span is one stop; the trace is the whole journey, and the parent-child links between spans show the call structure.

Concretely: a request hits your API gateway (span 1), which calls an auth service (span 2), which calls a database (span 3), then the gateway calls a pricing service (span 4). Those four spans, linked by one trace ID with parent-child relationships, render as a waterfall where the width of each bar is its duration. The widest bar is your problem.

Should I use Jaeger or OpenTelemetry?

Use both, because they do different jobs. OpenTelemetry is the vendor-neutral standard for instrumenting your code and exporting telemetry; Jaeger is a backend that ingests, stores, and visualizes traces. The modern pattern is to instrument with OpenTelemetry and export to Jaeger, which gives you Jaeger’s UI without coupling your code to it.

The practical implication is to standardize on OpenTelemetry for instrumentation across every service and language, then treat the tracing backend as a swappable detail. In a polyglot system this matters even more, because OpenTelemetry gives you one instrumentation model across all your languages instead of a different client per runtime.

What should you instrument?

Trace the boundaries first: every inbound request, every outbound call to another service, every database query, and every external API call. Those edges are where time is spent and where one service hands off to another, so they are where a trace earns its value. Propagating the trace context across each boundary is what keeps the spans stitched into one trace.

The non-negotiable is context propagation. A trace only works if the trace ID travels with the request across every hop, which means each service must read the incoming trace context and pass it onward. This is the same cross-service plumbing that, done wrong, produces the boundary failures in Why Language Boundaries Break Polyglot Microservices; propagation that silently drops at one language boundary leaves you with broken, half-length traces.

What you should not do is instrument every function. The instinct on adoption is to add spans generously, and it produces traces with hundreds of spans that are slower to read than the logs they replaced, while multiplying storage cost. A span earns its place if its duration could plausibly be the answer to “where did the time go.” An in-process function call that takes microseconds cannot be that answer; a network call or a disk read can.

The practical rule: span the things that can block. Network calls, database queries, cache lookups, disk I/O, lock acquisition, and queue publishes. Everything else is a candidate for a span attribute on the enclosing span rather than a span of its own. If a trace of a typical request does not fit on one screen, you have over-instrumented, and the cost is paid twice — in storage and in the seconds an on-call engineer spends scrolling during an incident.

How much should I sample traces?

Sample based on what you need to see, not a flat percentage. Tracing every request is expensive in storage and overhead at scale, but naive head-based sampling (decide at the start, keep 1%) will usually throw away the rare slow or failed request you most need. Tail-based sampling decides after the full trace is collected, so you can keep all the errors and slow traces and drop the boring fast ones.

The sampling tradeoff is the part teams underestimate, and it directly determines whether tracing helps during an incident.

• Head sampling (fixed rate): cheap and simple, but it decides before it knows whether the trace is interesting. The 1-in-100 you kept is probably a normal request; the slow one got dropped.
• Tail sampling: buffers the whole trace, then keeps it if it errored, exceeded a latency threshold, or is otherwise notable. More infrastructure, far better signal.

For a system where you are tracing to catch the rare bad request, tail-based sampling is usually worth the extra moving parts, because it guarantees the traces you actually open are the ones that matter.

The full decision, laid out:

	Head sampling	Tail sampling
When the decision is made	At trace start, before anything is known	After all spans are collected
Keeps errors and slow traces	Only by luck	By policy, reliably
Infrastructure needed	None beyond the SDK	A collector that buffers complete traces in memory
Cost profile	Lowest	Higher: buffering plus the collector tier
Failure mode	The trace you need was never recorded	Buffer pressure drops traces under load spikes
Good fit	Very high volume, uniform traffic, cost-dominated	Debugging rare failures; anything with an SLO

A pragmatic middle ground exists and is underused: head-sample a small baseline for traffic-shape visibility, and tail-sample aggressively for errors and latency outliers. You keep a representative sample of normal behavior cheaply, and you keep every trace that is actually interesting. If you only implement one policy, make it “keep 100% of errors and anything over your p99 threshold.”

One caveat that bites teams adopting tail sampling: the collector must see every span of a trace to make its decision, which means spans from one trace cannot be load-balanced across collector instances arbitrarily. They must be routed consistently by trace ID. Getting this wrong produces partial traces that look like broken context propagation, and teams debug the wrong problem for a week.

How do you read a trace waterfall?

Tracing tools are usually explained up to the point where a waterfall appears on screen, and then stop — as if the diagnosis were obvious. It is not, and the shapes are learnable. Four patterns cover most of what you will see.

One wide bar, everything else thin. The simplest case: a single downstream call dominates. Open that span, look at its own children. If it has none, the time is inside that service or its database, and you have narrowed the search to one team.

A staircase. Spans starting one after another, each waiting for the previous, forming a descending diagonal. This is serial execution of calls that are often independent. The fix is usually concurrency rather than making any individual call faster — five 40 ms calls in sequence is 200 ms; in parallel it is 40 ms.

A gap with nothing in it. Dead time between a parent span starting and its first child. Nothing is executing, which means the request is waiting: connection-pool exhaustion, thread-pool queueing, or a lock. This is the most commonly misread shape, because the instinct is to look for a slow service, and there isn’t one. The service is idle and blocked. It is also the shape that means you are near a capacity limit, so treat it as an early warning rather than a curiosity.

A bar wider than the sum of its children. Time is being spent in the service itself — serialization, in-process computation, garbage collection — not in anything it called. Worth checking against GC pause metrics before assuming it is application code.

The habit worth building: before diving into any single span, look at the shape first. The shape tells you which of the four problems you have, and each has a different class of fix.

What span attributes should you add?

Spans are only as useful as the questions you can ask of them, and that is decided by their attributes. The goal is to be able to answer “was this slow for a particular tenant, endpoint, or version?” without opening individual traces.

Attach the things you will filter by: the route template, the response status, the tenant or account identifier, the service version or build SHA, and the region or availability zone. Version is the one most often skipped and most often wanted — “did latency change after the Tuesday deploy” is a question you can only answer if the deploy is recorded on the span.

Two hard rules. Never put unbounded values in attributes you intend to aggregate on — a full URL with query parameters, or a raw user-supplied string, creates the same cardinality explosion that ruins metrics systems, a failure mode covered in Taming Prometheus Cardinality. Use the route template (/orders/{id}), never the concrete path. And never attach personal data or secrets: traces are widely readable inside an org, retained for weeks, and shipped to a third-party backend more often than not. Emails, tokens, and full request bodies do not belong in a span.

When do you not need distributed tracing?

Tracing is genuinely valuable, which makes it worth being clear about when it is not the right first investment — because installed-and-untrusted tracing is a common and expensive outcome.

If you run a monolith or two or three services, a profiler and good structured logs with a correlation ID will answer most of your questions at a fraction of the operational cost. The value of tracing rises sharply with the depth of your call graph, not the size of your traffic. A high-traffic system with a shallow graph needs metrics far more than it needs traces.

If your incidents are dominated by outright failures rather than latency, error tracking and alerting pay back faster. Traces shine when something is slow, and slowness is hard to attribute. A service that is down announces itself.

And if you cannot yet answer “what is our p99, and what should it be,” tracing will not help. Tracing tells you where time went inside a request you have already decided was too slow. Without an SLO to define “too slow”, you have a beautiful waterfall and no threshold to compare it against.

The honest sequencing for a young system is: structured logs with correlation IDs, then RED metrics per service, then SLOs, then tracing. Teams that install tracing first usually have the waterfall and still cannot tell whether what they are looking at is normal.

What is the performance overhead of tracing?

Small per request, but real in aggregate, which is exactly why sampling exists. Creating and exporting spans adds a little CPU and memory per request, and shipping every trace at high traffic adds meaningful network and storage cost. The instrumentation overhead is rarely the problem; the volume of trace data is.

This reframes overhead as two separate concerns. The in-process cost of generating spans is typically negligible relative to the work the request is already doing, so you do not avoid tracing to save CPU. The export and storage cost, however, scales with how many traces you keep, and that is where naive “trace everything” gets expensive fast at high request rates.

The resolution is the sampling decision from the previous section, plus exporting asynchronously so tracing never sits in the request’s critical path. Batch and export spans off the hot path, sample intelligently so you store the interesting traces and drop the rest, and the overhead becomes a rounding error against the debugging time it saves. Tracing you turned off to save a few percent CPU is tracing you do not have during the incident that would have paid for it many times over.

Why are my traces broken or incomplete?

A trace that ends abruptly three services in is the most common tracing failure, and it is almost always context propagation rather than a backend problem. The trace ID has to survive every hop, and there are a small number of places it reliably gets dropped.

Asynchronous boundaries. The classic. A request produces a message to a queue or an event bus, a consumer picks it up later, and the trace ends at the producer. Context does not ride along automatically — you must inject the trace context into the message headers on publish and extract it on consume. Until you do, every async path in your system is a trace terminator. This is worth fixing early, because async hops are exactly where hard-to-debug latency hides.

Thread and task handoffs. Passing work to a thread pool, a background task, or a new goroutine can lose the ambient context depending on the language and instrumentation. The span appears to end when the request handler returns, while the real work continues untracked.

Unenrolled services. One service in the chain has no instrumentation, or has it misconfigured. It receives the trace headers and does not forward them, silently severing everything downstream. In a polyglot system this is disproportionately likely at the boundary of whichever language got instrumented last — the same class of seam failure described in Why Language Boundaries Break Polyglot Microservices.

Header stripping at a proxy. Load balancers, API gateways, and service meshes can drop unknown headers depending on configuration. If traces break at exactly the same hop every time and the service looks correctly instrumented, check what sits in front of it.

Clock skew. Not a break, but it looks like one: spans render with impossible timings, children starting before parents, or negative durations. The trace is intact; the machine clocks disagree. Run NTP everywhere and treat wild timings as an infrastructure signal rather than an application bug.

The way to catch all of these is a synthetic end-to-end trace check: a low-rate request that deliberately traverses the deepest path in your system, with an assertion that the resulting trace contains the expected number of hops. Alert when it does not. That converts “our tracing is broken” from something you discover mid-incident into something you learn on a Tuesday afternoon.

A tracing setup checklist

Before you rely on Jaeger in an incident:

• Instrumentation is OpenTelemetry-based, not backend-specific.
• Trace context propagates across every service boundary, verified end to end (no broken traces).
• Inbound requests, outbound calls, DB queries, and external APIs are all spanned.
• Sampling keeps errors and slow traces (tail-based if you can run it), not just a flat percentage.
• Spans carry useful tags (status codes, key IDs) without leaking sensitive data.
• Trace retention and storage cost are sized deliberately; traces are voluminous.
• The team knows how to go from a latency alert to the relevant trace quickly.

What I’d do differently

The mistake I have made is treating tracing as a checkbox: install it, see a few traces in the demo, move on. Then an incident hits, you open Jaeger, and the trace you need was sampled away, or it ends abruptly because context propagation broke at one service. Tracing that you have not validated under real conditions is tracing you cannot trust when it counts.

If I were rolling out tracing again, I would validate two things before declaring it done: that a trace survives end to end across every service without breaking, and that the sampling strategy actually keeps slow and failed requests. Get those right and Jaeger becomes the first place you look during a latency incident. Get them wrong and it becomes a dashboard you stop trusting. The dashboards that complement it are the subject of Grafana Dashboards for Operators, Not Executives.

Sources

• Jaeger, Architecture: jaegertracing.io/docs/latest/architecture
• OpenTelemetry, Traces: opentelemetry.io/docs/concepts/signals/traces
• OpenTelemetry, Sampling: opentelemetry.io/docs/concepts/sampling