Taming Prometheus Cardinality

The thing that kills Prometheus is almost never query volume. It is cardinality: the number of unique time series it has to hold in memory, which explodes the moment someone puts an unbounded value like a user ID or a request ID into a label. Each unique label combination is a separate series, so one careless label can turn a handful of metrics into millions of series and take the server down. Taming Prometheus is mostly a discipline of what you are allowed to put in a label.

This is the single most common way teams break their own metrics stack, and it is subtle because the offending line of code looks completely reasonable. Adding a label feels free. It is not.

Why cardinality is the metric that matters

Prometheus stores each unique time series in memory, and a time series is defined by a metric name plus the exact set of label key-value pairs on it. http_requests_total{method="GET", status="200"} and http_requests_total{method="GET", status="500"} are two separate series. Your total series count, your cardinality, is what drives Prometheus memory use and, past a point, its collapse.

This reframes the whole cost model. Adding a new metric name is cheap. Adding a label whose values are unbounded is what multiplies your series count without limit, and memory is what runs out. This post is part of the Observability series, alongside Jaeger tracing and operator dashboards.

What causes a cardinality explosion in Prometheus?

A cardinality explosion happens when you put a high-cardinality value into a label: a user ID, request ID, session ID, email, full URL with query parameters, timestamp, or any value that can take many distinct forms. Because every distinct value creates its own time series, a single such label multiplies your series count by the number of distinct values it sees, which for something like user ID is effectively your entire user base.

The insidious part is that these labels are added for good reasons, someone wants to slice requests by user, or see a specific request, and a label seems like the way. It is exactly the wrong tool: that per-entity detail belongs in logs or traces, which are designed to hold high-cardinality data, not in metrics, which are designed for bounded aggregates.

How do you fix high cardinality in Prometheus?

Remove or bound the offending labels. Replace unbounded values with low-cardinality ones: instead of the full URL, use the route template (/users/:id); instead of a user ID, drop it from the metric entirely and rely on logs for per-user detail. Aggregate away dimensions you do not actually query on, and move genuinely high-cardinality needs to the systems built for them.

The rules that keep cardinality bounded:

• Labels must have bounded value sets. Status code, HTTP method, route template, region, service name: all bounded. IDs, URLs, timestamps, emails: never.
• Use route templates, not raw paths. /users/:id is one label value; /users/12345 is a million.
• Multiply before you ship. The series count for a metric is the product of its labels’ distinct values. Two labels with 10 and 20 values is 200 series; add a 10,000-value label and it is 2,000,000. Do that arithmetic before adding a label.
• Send per-entity detail to logs and traces. Structured logs and traces are built to hold user IDs and request IDs. Metrics are not.

Bad label (unbounded)	Good label (bounded)
user_id, customer_id	(omit; use logs/traces)
full URL with query string	route template (/api/:resource)
request_id, session_id	(omit; correlate via traces)
raw error message	error type / class
exact timestamp	(never a label)

How much cardinality can Prometheus handle?

It scales with memory, because Prometheus keeps active series in memory, so a well-provisioned server handles on the order of millions of active series, but the right goal is not to chase a maximum. It is to keep cardinality bounded and predictable so it cannot grow without limit as your traffic and user base grow. A metric whose cardinality is a fixed product of bounded labels is safe; one whose cardinality tracks your user count is a future outage.

The practical stance is to treat cardinality as a budget. Know roughly how many active series each service contributes, watch that number, and make sure no metric’s series count is tied to an unbounded dimension. A stack that stays well within its memory with predictable, bounded cardinality is healthy; one that is one careless label away from doubling is not, regardless of how much headroom it has today.

How do you monitor and limit cardinality?

Watch your series count and set limits so a bad deploy cannot silently explode it. Prometheus exposes its own metrics about how many series it is holding and which metrics contribute most, and you can query for the highest-cardinality metrics directly. Sampling that regularly, and alerting when total series or per-metric series jump, turns a cardinality explosion from a silent memory creep into an early warning.

Beyond monitoring, enforce limits. Prometheus supports sample and label limits per scrape target, which cap how many series a misbehaving target can introduce, so one bad service cannot take down the whole server. Combine a per-target limit (the safety net) with a review habit (every new label gets the “is this bounded?” question) and cardinality stays controlled by policy rather than by luck. This is the same defensive posture as putting an acyclicity check in CI: make the dangerous thing hard to do by accident.

What do you do when cardinality is already too high?

Find the worst offenders, then drop or relabel them, because you rarely need to fix everything at once. Prometheus can tell you which metrics have the most series, so you attack the top few first, the ones contributing the bulk of the load, and get most of the relief from a handful of changes. Cardinality is almost always dominated by a small number of bad metrics, not spread evenly.

The recovery sequence when a server is straining or has fallen over:

• Identify the top-cardinality metrics. Query for the metrics with the most series; the distribution is usually very skewed, so a few are responsible for most of the problem.
• Find the offending label on each. It is nearly always one unbounded label (an ID, a URL, a raw message) doing the damage.
• Drop or rewrite it at the source, or relabel at scrape time. You can stop emitting the label, collapse it (URL to route template), or use relabeling rules to drop it before ingestion as an immediate stopgap.
• Add a per-target limit so the same metric cannot re-explode after you fix it.

Relabeling at scrape time is the fast mitigation: you can drop a runaway label in the scrape config without waiting for an application deploy, which buys you time to fix the instrumentation properly. Then fix it at the source, because dropping at scrape time still pays to transfer the data. The durable fix is always removing the bad label where it is emitted.

What do you do when cardinality is already out of control?

Prevention is the previous section. This one is for the situation where Prometheus is already struggling and you need it back today.

Work in this order, because it moves from reversible to permanent:

1. Drop the offending label at scrape time. A metric_relabel_configs rule with action: labeldrop removes it before ingestion. This is a config change, applies on reload, and needs no application deploy — which makes it the correct first move even if you plan a proper fix later.
2. Drop the whole metric if it is not load-bearing. Some metrics are instrumented and never queried. action: drop on the metric name costs nothing to try and is trivially reversible.
3. Cap the target with a sample_limit. This prevents one misbehaving exporter from taking the server down again while you work, converting a monitoring outage into a single failed target.
4. Reduce retention temporarily. Buys memory and query headroom immediately at the cost of history, and is easy to restore.
5. Fix the instrumentation. The permanent repair, but it requires a deploy and therefore is not the emergency lever.

The point of that ordering: the first three are edits to Prometheus configuration, not to your services. During an incident you want changes you can make and reverse in minutes without a release cycle, and cardinality is unusual among production problems in that a config-only mitigation genuinely exists.

Two things to know about recovery. Dropping a label does not immediately reclaim memory — the existing series stay in the head block until they age out and compaction runs, so expect improvement over minutes to hours rather than instantly. And the dropped series remain queryable historically, which is occasionally useful and occasionally confusing when someone queries an old range and sees dimensions that no longer exist.

Finally, treat the recovery as incomplete until there is a guard. A cardinality incident that gets mitigated without adding a sample_limit, an alert on prometheus_tsdb_head_series growth, or a review rule for new labels will recur, because the instrumentation habit that caused it is unchanged. The fix is cheap and the recurrence is expensive.

What belongs in a label, and what does not?

Most cardinality incidents are prevented by one decision made correctly at instrumentation time, so it is worth having an explicit rule rather than an instinct.

A value belongs in a label only if the set of possible values is small, stable, and something you would group or filter by. All three conditions, not two.

Candidate	Label?	Reasoning
HTTP status code	Yes	Tiny, fixed set; you always group by it
Route template (/orders/{id})	Yes	Bounded by your route count
Concrete path (/orders/8f2c…)	No	Unbounded — one series per order, forever
Service, region, environment	Yes	Small and stable
Deployment version	Careful	Useful, but churns on every release; bound the retention
Pod name / instance ID	Careful	Churns constantly; drop it unless you truly need per-pod views
User ID, tenant ID, order ID	No	Unbounded by definition
Error class (timeout, validation)	Yes	A small enumerated set you choose
Error message	No	Effectively unbounded free text
Email, IP address, session ID	No	Unbounded, and a privacy problem in telemetry

The test that resolves ambiguous cases: could this label’s value set grow with your traffic? If more users, orders, or requests mean more distinct values, it is unbounded, and unbounded values do not belong in metrics regardless of how useful they would be.

That last clause is where the real discipline lies. The reason engineers add user_id to a metric is that they genuinely want to answer “is this slow for a particular customer” — a legitimate question with a wrong tool. Metrics answer aggregate questions; traces and logs answer per-entity ones. Put the customer identifier on a span or a log line, where the storage model expects high cardinality, and the question gets answered without threatening the monitoring system. The same reasoning governs span attributes, and it is the single most useful boundary to internalise across all three signals.

A practical guardrail worth adding early: enforce a per-metric series limit in the scrape config. Prometheus can be configured to refuse a target that exceeds a sample limit, which converts a cardinality explosion from “the monitoring system dies” into “one target stops being scraped and an alert fires.” That is a far better failure mode, and it turns a potential outage into a ticket.

How do you find which metric is causing the problem?

Knowing you have a cardinality problem is easy — memory climbs, queries slow, Prometheus eventually gets OOM-killed. Finding which of your two thousand metrics is responsible is the actual work, and there is a reliable order to it.

Start with Prometheus’s own instrumentation, which answers most cases in a minute:

• prometheus_tsdb_head_series — total active series. This is the headline number to trend; a step change in it dates the problem to a deploy.
• topk(10, count by (__name__)({__name__=~".+"})) — the ten metric names with the most series. Usually one metric is responsible for a large share of the total, and this query names it immediately.
• /api/v1/status/tsdb — the built-in endpoint reporting top series by metric name, top label names by cardinality, and top label-value pairs. This is the single most useful diagnostic and the most overlooked.
• count(count by (label_name)(your_metric)) — once you have the metric, this tells you which label is doing the damage.

The pattern that emerges is almost always the same: one metric, one label, one dimension that someone added because it seemed useful. A user_id, a request_id, a raw URL path, an error message string, an email address. Each unique value creates a permanent new time series.

The arithmetic is what makes this dangerous. Cardinality multiplies across labels rather than adding. A metric with 5 status codes × 20 endpoints × 3 regions is 300 series, which is fine. Add a customer_id label with 10,000 values and it becomes 3,000,000 series from a single metric. Nothing about the change looks alarming in review — it is one label — and it can take down the monitoring system that would have told you about it.

Why is a churning label as bad as a large one?

There is a second, subtler form of the problem that the counts above can hide: high churn.

A label whose values change constantly — a pod name in a frequently-redeployed service, a version string, a build ID, a session identifier — may show modest active cardinality at any instant while generating enormous total series over a retention window. Each pod restart creates a new series that stays queryable until retention expires.

This produces a confusing signature: active series looks acceptable, but memory grows steadily, queries over longer ranges get slower and slower, and compaction takes longer each cycle. Teams look at the current series count, conclude cardinality is fine, and go looking for the wrong problem.

The rule that covers both cases: a label is safe only if its set of possible values is small and stable. Bounded but churning is still unbounded over time. Pod name is the canonical offender, and it is usually added automatically by an exporter rather than deliberately by an engineer — which is why relabelling rules that drop it are one of the highest-value configuration changes available.

The mitigations follow directly. Drop churning labels at scrape time with metric_relabel_configs, since dropping at ingestion costs nothing and is reversible. Prefer a stable identifier — deployment or service name — over an ephemeral one. And keep the identifying detail in logs and traces, where high cardinality is the expected shape and the storage model is built for it, rather than in metrics, where it is the one thing the storage model cannot absorb.

A cardinality-control checklist

Before you add a label or ship a new metric:

• Every label has a bounded, enumerable set of possible values.
• No label carries an ID, full URL, raw message, email, or timestamp.
• Raw paths are collapsed to route templates.
• You did the multiplication: metric series count = product of label value counts, and it is bounded.
• Per-entity detail goes to logs/traces, not metric labels.
• Per-target sample/label limits are set so one service cannot explode the server.
• You monitor total active series and alert on sudden growth.

Worth stating plainly at the end: cardinality is the one Prometheus failure mode that scales with your success. More customers, more endpoints, and more services all push it upward, so a configuration that was comfortable at launch will not stay comfortable. Treat the series count as a capacity metric you trend and plan against, in the same way you would trend disk or memory, rather than as a number you look at only when something has already broken.

What I’d do differently

The mistake I have watched (and made) is adding a label in a moment of “I just want to slice by this one thing,” without asking whether that thing is bounded. Metrics feel free to extend, and that instinct is exactly what detonates a Prometheus server weeks later when the label’s value set grows with the business.

If I were setting metric conventions from scratch, I would make “is this label bounded?” a required question in code review, and wire a per-target series limit as a hard backstop so no single mistake can take the whole stack down. High-cardinality data is not forbidden; it just belongs in logs and traces, which is why the observability stack has three pillars. Metrics are for bounded aggregates. Keep them that way and Prometheus stays boringly reliable, which is the only kind of monitoring worth having.

Sources

• Prometheus, Metric and label naming best practices: prometheus.io/docs/practices/naming
• Prometheus, Instrumentation practices (avoiding high cardinality): prometheus.io/docs/practices/instrumentation
• Prometheus, Storage and memory use: prometheus.io/docs/prometheus/latest/storage