OpenTelemetry Sampling Strategies

OpenTelemetry sampling is the decision that determines whether your tracing is useful when it matters. Tracing every request at scale is too expensive in overhead and storage; tracing 1% at random is cheap and will throw away the exact slow or failed request you needed to debug. The whole art is keeping a small fraction of traces while making sure that fraction includes the interesting ones. That is the difference between head sampling (decide up front, blind) and tail sampling (decide after, informed).

This is the part of tracing that teams underinvest in, and it is why so many teams “have tracing” and still can’t find the trace during an incident.

Why sampling decides whether tracing is worth it

Distributed tracing’s value is letting you open the trace for a slow or failed request and see which service caused it. That value evaporates if the trace you need was never recorded. Sampling is what stands between “we have tracing” and “we have the trace that explains this incident,” and getting it wrong is the quiet reason tracing disappoints.

The tension is real: keeping every trace is genuinely expensive at high request rates, both in per-request overhead and in storage. So you keep a fraction. The entire question is which fraction, and that is what head versus tail sampling answers. This post is part of the Observability series and builds on Jaeger Tracing for Cross-Service Debugging.

What is the difference between head and tail sampling?

Head sampling decides whether to keep a trace at its very beginning, before the request has run, typically by keeping a fixed percentage. Tail sampling buffers the entire trace until it completes, then decides based on how it turned out, so it can deliberately keep every error and every slow trace and drop the ordinary fast ones. Head is cheap and simple but blind to the outcome; tail is smarter but requires buffering infrastructure.

	Head sampling	Tail sampling
When it decides	At trace start	After trace completes
Basis	Fixed rate / trace ID	Actual outcome (errors, latency)
Keeps the rare bad trace?	Only by luck	Yes, on purpose
Cost / complexity	Low	Higher (buffer whole traces)
Best for	High volume, cost-first	Debuggability-first, incident readiness

The reason this distinction dominates the topic is that it maps directly to whether you will have the trace you need. Head sampling optimizes cost; tail sampling optimizes usefulness. Most mature setups lean toward tail sampling for exactly that reason.

What sampling rate should I use?

Stop thinking in terms of a single rate and think in terms of what you keep. With head sampling, a low percentage saves the most cost but drops rare important traces; a high percentage keeps more but costs more, and still keeps interesting and boring traces in the same proportion. With tail sampling you can keep a small fraction of total traffic while retaining essentially all errors and slow requests, which is almost always what you actually want.

So the target is a policy, not a number: keep all the failures, keep the slow tail, sample the routine baseline lightly for capacity trends. That configuration keeps your storage small and your incident-time trace lookups almost always successful, which is the entire point of running tracing.

Does sampling lose important traces?

Head-based random sampling does, structurally, because it decides before the request’s outcome is known, so the rare slow or failed request is statistically likely to be among the dropped majority. Tail-based sampling is specifically designed to avoid this: by waiting until the trace finishes, it keeps errors and high-latency traces on purpose. If losing the interesting trace during an incident is unacceptable, and it usually is, that is the argument for tail sampling.

There is a subtlety worth knowing: sampling decisions must be consistent across a whole trace, or you get broken partial traces where some services in the request kept their spans and others dropped them. OpenTelemetry propagates the sampling decision as part of the trace context so the whole trace is kept or dropped together. This is the same context-propagation discipline that keeps traces intact across language boundaries; if propagation breaks, sampling breaks with it.

How do you implement tail sampling in OpenTelemetry?

Run the OpenTelemetry Collector with a tail-sampling processor. Because the decision needs the complete trace, it happens in the Collector (not in each service), which buffers spans until the trace finishes and then applies your policies, keep on error, keep on latency over a threshold, keep a percentage of the rest. Services do head-sample-nothing (or minimally) and export everything to the Collector, which makes the smart decision centrally.

The operational cost to be aware of is that tail sampling requires the Collector to hold spans in memory until traces complete, which needs enough memory and careful handling when many Collector instances see different spans of the same trace. This is the “higher infrastructure” cost of tail sampling in concrete terms, and it is why some teams start with head sampling and graduate to tail sampling once debuggability during incidents becomes the priority. Either way, standardize on the OpenTelemetry Collector so the sampling policy lives in one place, not scattered across services.

Can you combine head and tail sampling?

Yes, and mature setups often do. A light head sample at the service reduces the raw volume the collectors have to buffer, and tail sampling in the collector then makes the smart keep-the-interesting-ones decision on what remains. Used carefully, this balances the cost of buffering everything for tail sampling against the risk of head sampling dropping too much too early.

The nuance is to head-sample gently if at all, because anything head sampling drops is invisible to the tail sampler downstream, no matter how interesting it would have turned out to be. If you head-sample at 10% and then tail-sample, you can only ever keep interesting traces from that 10%; the interesting trace in the discarded 90% is gone forever. So the common pattern is to keep head sampling near 100% (drop little or nothing early) and let tail sampling do the real reduction based on outcome.

The exception is extreme volume where buffering 100% for tail sampling is genuinely infeasible. There, a modest head sample is a pragmatic tradeoff: you accept losing some rare events in exchange for making tail sampling affordable at all. That is a deliberate cost decision, not a default, and you should make it knowing exactly what it gives up, which is guaranteed capture of the rarest failures. For most systems, near-full head sampling plus outcome-based tail sampling is the right combination.

A sampling-strategy checklist

When configuring OpenTelemetry sampling:

• You have decided based on outcome (tail) versus cost-first (head), deliberately.
• If debuggability matters, tail sampling keeps 100% of errors and slow traces.
• The routine baseline is sampled lightly for trend/capacity, not kept in full.
• Sampling decisions are consistent across the whole trace (no broken partial traces).
• Trace context propagates across every service and language boundary.
• Tail sampling runs in the Collector with enough memory for in-flight traces.
• You verified, under a simulated incident, that the slow/failed trace is actually retained.

What does tail sampling cost to run?

Tail sampling is the right answer for most teams and it is not free, and the costs are structural rather than incidental.

The collector must buffer complete traces in memory. It cannot decide until every span of a trace has arrived, which means holding all spans for the duration of the decision window. Memory usage scales with trace throughput multiplied by trace duration, and long traces are disproportionately expensive to hold.

All spans of one trace must reach the same collector. This is the constraint that surprises people. You cannot load-balance spans arbitrarily across collector replicas; they must be routed consistently by trace ID, which usually means a two-tier collector deployment with a load-balancing layer in front. Getting this wrong produces partial traces that look exactly like broken context propagation, and teams debug the wrong problem for days.

Late spans are lost. The decision window has to end sometime. A span arriving after its trace was evaluated is dropped, so a slow service at the end of a chain can be systematically missing from the traces you keep — precisely the service you most wanted to see.

Under memory pressure, the collector drops traces. The failure mode is silent and it correlates with load, meaning you lose traces exactly when the system is busiest.

The mitigations are straightforward once the costs are visible. Size the decision window against your realistic p99 trace duration, not your median. Alert on collector memory and on dropped-span counters. And treat the collector tier as production infrastructure with its own capacity plan, because it now sits between you and all of your trace data.

What sampling policies are actually worth configuring?

A policy set that covers most needs, in priority order:

• Keep 100% of traces containing an error. Non-negotiable, and the single highest-value rule.
• Keep 100% of traces exceeding a latency threshold. Set the threshold from your SLO, so the traces you keep are the ones that breached it.
• Keep a small baseline of normal traces — a percent or two — so you retain a picture of healthy behaviour for comparison. Debugging a slow trace is much harder without a fast one to compare against.
• Keep more traces from low-volume endpoints. A uniform rate means a rarely-used endpoint may produce no traces at all, and rare endpoints are frequently where bugs hide.
• Keep traces for specific tenants or users on demand, when investigating a reported problem.

The last two are what separate a thoughtful policy from a basic one, and the low-volume rule is the more valuable. Sampling proportionally to traffic means your busiest endpoint is thoroughly observed and your obscure ones are invisible — which inverts where you actually need visibility, since the busy path is the one everyone already understands.

How does sampling interact with metrics and logs?

Sampling decisions are usually made about traces in isolation, which produces two failure modes worth designing around.

Do not compute metrics from sampled traces. If you keep 1% of traces and count errors from them, your error count is wrong by a factor of a hundred — and worse, it is wrong by an unknown factor once tail sampling makes the rate non-uniform, because you deliberately keep all errors and few successes. Any ratio derived from that data is meaningless. Metrics must come from unsampled counters emitted by the application; traces answer “what happened in this request,” not “how many.”

This is the most common analytical error in observability setups, because a trace backend will happily show you a chart of error counts and it looks authoritative.

Keep logs and traces consistent. If a trace was kept, its logs should be findable, and the join is the trace ID. Where the two are sampled independently, you get the frustrating case of an interesting trace whose logs were discarded — or expensive logs retained for traces nobody kept. The workable pattern is to sample logs by trace decision where your tooling allows it, so the two stay aligned.

A related capability worth building: trace-scoped debug logging. When a trace is selected for retention, raise its log verbosity for that request only. You get full detail for exactly the requests you kept and normal volume everywhere else, which is the most cost-effective debugging arrangement available and requires only that the sampling decision be visible to the logger.

The general principle: the three signals should agree about which requests are interesting. When metrics, traces, and logs each make independent retention decisions, investigating an incident means reconciling three partial views, and the reconciliation is exactly the work you were trying to avoid.

How do you know your sampling is working?

Sampling is configured once and rarely verified, which is how teams discover during an incident that the traces they needed were never kept. Four checks make it observable.

Track the effective sampling rate, by outcome. Not one number — separate rates for errors, slow requests, and normal traffic. The headline “we sample 5%” tells you nothing if errors are being dropped. If your error retention is not effectively 100%, the policy is not doing what you think.

Alert on dropped spans at the collector. Every collector exposes counters for spans it discarded due to memory pressure or queue overflow. A non-zero value under load means you are losing traces exactly when they matter, and it is invisible unless you look.

Run a synthetic trace check. A low-rate request that deliberately traverses your deepest path, with an assertion that the resulting trace is retained and contains the expected number of hops. This catches both broken sampling and broken context propagation, and it catches them on a Tuesday rather than during an outage.

Review after every incident. The postmortem question is simply: did we have the trace we needed? If not, was it sampled away, was context propagation broken, or was the service not instrumented? Each has a different fix, and each is cheap to make once identified.

The underlying point is that sampling configuration is a reliability-affecting setting that silently degrades. A new service joins with default head sampling. A collector’s memory limit stops matching trace volume. A policy threshold set against last year’s latency no longer selects anything. None of these produce an error; they produce an absence, and absences are only noticed when someone goes looking for something that should have been there.

Should sampling ever be dynamic?

Static sampling is the right default and there are two cases where making it adjustable pays for itself.

During an incident, you want more traces. The rate that is economical at steady state is frequently too sparse when you are actively debugging. Being able to raise retention for a specific service, endpoint, or tenant for an hour — without a deploy — turns tracing from a passive record into an investigative tool. This is worth building before you need it, because building it mid-incident is not an option.

For a specific user report, you want targeted capture. “Customer X says checkout is slow” is much easier to answer if you can retain 100% of that tenant’s traces temporarily. A tenant-scoped override, applied at the collector, costs little and answers a class of support question that is otherwise nearly unanswerable.

What to avoid is automatic rate adjustment based on volume, which sounds sensible and behaves badly. A collector that lowers the sampling rate as traffic rises reduces visibility precisely as the system gets busier — the opposite of what an operator wants — and it makes retained-trace counts non-comparable over time, so you can no longer tell whether errors increased or sampling changed.

The rule that keeps this sane: make the sampling rate adjustable by a human, and never by a feedback loop. Deliberate temporary overrides are valuable; a system that quietly changes its own observability under load is one you cannot reason about at exactly the moment reasoning matters.

One practical closing note on rollout: change sampling policy in one service first, verify the effect, then propagate. Sampling is global-feeling configuration that is easy to apply fleet-wide in a single commit, and a policy that looks correct can turn out to retain far more or far less than intended once real traffic shapes hit it. A staged rollout costs a day and prevents both the surprise bill and the surprise blind spot. Watch the effective retention rate by outcome during that day rather than the raw trace count, since the raw count can look stable while the mix underneath it has shifted entirely.

The habit that makes all of this sustainable is to treat sampling configuration as code that is reviewed like any other production change, rather than as a collector setting someone adjusts ad hoc. Policies drift, thresholds age out, and the person who understood the original reasoning moves on — a reviewed, commented configuration file is what keeps the intent legible a year later.

What I’d do differently

The mistake I have made is treating sampling as a set-and-forget percentage, dropped in at setup and never revisited, then discovering during an incident that the trace I needed was one of the 99% thrown away. A flat head-sampling rate feels responsible and quietly guarantees you will miss the rare events that matter most.

If I were setting up tracing again, I would go to tail sampling as soon as the volume justified sampling at all, and configure it by interestingness: all errors, all slow traces, a thin sample of the rest. And I would test it the way I test a backup, by simulating the failure and confirming the trace survived. Tracing you have not verified under incident conditions is tracing you are only hoping works, and hope is not an observability strategy.

Sources

• OpenTelemetry, Sampling: opentelemetry.io/docs/concepts/sampling
• OpenTelemetry Collector, Tail sampling processor: opentelemetry.io/docs/collector
• OpenTelemetry, Traces and context propagation: opentelemetry.io/docs/concepts/signals/traces