Kafka Partitions, Retention, and Compaction

Three settings on a Kafka topic decide most of what you will care about later: partition count sets your throughput ceiling and consumer parallelism, retention sets how far back you can replay, and cleanup policy decides whether the topic is a history of facts or the current state of the world. Get them right at creation and the topic is quietly correct for years. Get them wrong and you find out during an incident, when a replay cannot reach far enough back or a consumer group cannot scale past a partition count you cannot change.

None of these are defaults you should accept. Each is a decision with a distinct consequence, and each is significantly harder to change after the topic has traffic.

Why these three, specifically

A Kafka topic has many settings. These three are the ones with consequences you cannot easily undo and that shape how the rest of your system is allowed to behave.

Partition count is a structural decision: it determines how many consumers can work in parallel, and reducing it is not supported, so an over-provisioned topic is permanent overhead and an under-provisioned one is a scaling wall. Retention is a capability decision: it silently defines what recovery operations are even possible. Cleanup policy is a semantic decision: it determines whether your topic is a log of things that happened or a table of current values, which changes what consumers can correctly do with it.

Everything else is tuning. These three are architecture. This post is part of the Distributed systems patterns series and follows directly from topic naming and governance, since these are the fields your registry should declare for every topic.

How many partitions should a Kafka topic have?

Enough to support the consumer parallelism you actually need, and no more. The governing constraint is simple: within a consumer group, each partition is consumed by at most one consumer. So a topic with three partitions supports at most three working consumers in a group. Add a fourth consumer and it sits idle.

That makes partition count your parallelism ceiling. If you need to process a stream with twelve workers, you need at least twelve partitions, and no amount of autoscaling gets you past that number. This is a common and confusing scaling failure: the autoscaler adds replicas because queue depth is climbing, the new pods start, join the group, and do nothing, because there are no free partitions for them.

The asymmetry that should drive your default: adding partitions is easy; removing them is not. Kafka supports increasing partition count on a live topic. It does not support decreasing it. Worse, adding partitions changes the key-to-partition mapping, so events for a given key may land on a different partition than their history did, breaking per-key ordering across the boundary. Neither direction is free.

Given that, the sensible posture is to start small and grow deliberately, sizing from two inputs:

• Required throughput. Estimate per-partition throughput for your workload, then divide.
• Required consumer parallelism. How many workers do you need draining this stream at peak?

Take the larger, add modest headroom, and stop. Provisioning fifty partitions for a topic that carries a handful of events per minute costs you real overhead in broker file handles, replication traffic, rebalance time, and end-to-end latency, in exchange for parallelism you will never use.

What is the difference between delete and compact?

They answer different questions, and choosing between them is really choosing what your topic means.

cleanup.policy=delete removes records once they age past the retention window. The topic is a history of facts: things happened, they are recorded in order, and after some time the old ones stop being useful and are discarded. user.registered is a fact stream. So is race.finished and payment.captured.

cleanup.policy=compact keeps the most recent record for each key and eventually removes superseded records for that key. The topic is a table: for every key, here is the current value. Old values for a key are garbage once a newer one exists.

	delete	compact
What the topic represents	History of events	Current state per key
What ages out	Records older than retention	Superseded records for a key
Typical retention	Bounded window (days)	Infinite (-1)
Replay gives you	Events in a time window	Latest value for every key
Right for	Facts, audit trails, event streams	Changelogs, state, materialized views
Wrong for	Rebuilding current state from scratch	Anything where history is the point

The distinction is not stylistic. A compacted topic destroys history by design, that is its job. If you compact an audit trail, you have silently deleted the record of everything except the latest state per key, and you will discover this at exactly the moment someone asks what happened three months ago.

When should you use Kafka log compaction?

Use compaction for changelog and state topics, where what you want from the topic is the current value for every key rather than the sequence of changes. The discipline I hold is narrow and worth stating as a rule: compaction is only for changelog/state topics, and those carry infinite retention.

Two examples of topics that qualify. A subscription-state topic, where a consumer needs to know the current plan for every user; replaying it from the beginning yields exactly one record per user, the latest, which is precisely what rebuilding state requires. A leaderboard-state topic, where only the current standing per entity matters and prior standings are noise once superseded.

The pairing with infinite retention is deliberate, not incidental. Compaction’s value is that a brand-new consumer can read the topic from the beginning and reconstruct complete current state for every key. If the topic also had a time-based retention window, keys that had not been updated recently would age out entirely, and your “complete state” would be missing every quiet key. Compaction plus a retention window gives you a state topic with holes in it, which is worse than either choice alone.

So the rule pairs: compact + retention -1, together, for state topics only. Everything else gets delete with an explicit, chosen retention window.

How long should you retain Kafka data?

Long enough to cover two things: how far back you might need to replay, and the longest outage window during which a consumer might be down and unable to read.

The replay constraint is the one that surprises people, because it is absolute. Retention is your replay ceiling. A topic with seven-day retention supports at most a seven-day replay. If you need to rebuild a projection from the beginning of time, no retention window will do it; you need a compacted state topic or an external archive. This is the hard limit behind every Kafka replay strategy, and it is decided months earlier, when someone set retention without thinking about recovery.

The outage constraint is subtler and more dangerous. If a consumer is down for longer than the retention window, the events it never processed are gone. It comes back, resumes from the earliest available offset, and has silently skipped everything that expired while it was down. No error is raised. The consumer looks healthy and its state is permanently wrong.

That risk is why some topics deserve deliberately longer retention than their traffic would suggest. The clearest case is deletion and compliance events. If a user.deleted event expires before a consumer that was down comes back, that service never purges the user’s data, and you have a silent compliance violation that nothing will surface. On the platform I run, deletion-related events carry a substantially longer retention window than ordinary event topics for exactly this reason: no consumer should be able to miss a deletion across any realistic outage. The full emission contract behind that is its own subject, covered in GDPR Deletion in Event-Driven Systems.

The tiering that falls out of this:

• Ordinary fact streams: days. Long enough for a normal replay and a normal outage.
• Deletion, compliance, and audit-adjacent events: substantially longer, sized so no outage window can cause a miss.
• State/changelog topics: infinite, with compaction.

A worked sizing example

Making partition sizing concrete, because “size it from throughput and parallelism” is easy to say and vague to apply. Suppose you are declaring a topic for a moderately busy business event.

Step 1, required consumer parallelism. Your consumer takes roughly 20 ms per event and you expect peak sustained throughput of 400 events/second. One consumer handles ~50 events/second, so you need about 8 consumers working in parallel at peak. That sets a floor of 8 partitions, because a ninth consumer would idle.

Step 2, throughput headroom. Kafka partitions handle far more than 50 events/second, so throughput is not your binding constraint here; consumer processing time is. When the reverse is true (very high volume, very cheap processing), divide your peak byte throughput by a conservative per-partition throughput figure and take that instead.

Step 3, growth headroom, modestly. You might double traffic within the year, so 12 to 16 partitions leaves room without absurd over-provisioning. Beyond that you are buying rebalance latency and broker overhead for capacity you have no plan to use.

Step 4, sanity-check the key. With user_id as the key, are any users hot enough to skew one partition? A leaderboard event keyed by a single global leaderboard ID would put everything on one partition regardless of how many you declare. If your key is that skewed, more partitions will not help and the key is the thing to fix.

The takeaway is that partition count is derived from consumer economics, not from traffic alone. Most teams that over-provision did so by reasoning only about message volume and never about how long a consumer takes per message.

How do you change partition count safely?

You can increase partitions on a live topic, and you should treat it as a planned operation rather than a quick fix, because of the key-mapping consequence.

Kafka assigns keys to partitions by hashing the key modulo the partition count. Change the count and the modulus changes, so existing keys start mapping to different partitions. New events for a user may land on a different partition than that user’s history. Within a partition, order is preserved; across the change boundary, per-key order is not.

Whether that matters depends entirely on your consumers. A consumer that treats each event independently, counting, notifying, forwarding, does not care. A consumer that builds per-entity state by folding events in order can be corrupted by seeing a later event before an earlier one it never received on the new partition.

The safe sequence when order matters:

1. Confirm which consumers depend on per-key ordering. If none do, just increase it.
2. If some do, prefer draining to a quiet period so the reordering window contains little or no traffic.
3. Consider whether a new topic with the desired partition count plus a migration is cleaner than mutating in place; for critical ordered streams it often is.
4. Increase, then verify consumer lag and per-key behavior before considering it done.

And never plan to reduce. It is not supported, and the workaround (create a new topic, migrate, delete the old) is enough work that it is better to avoid over-provisioning in the first place.

Retention, storage cost, and the tiered-storage escape

Retention is a direct storage bill: bytes per second, times seconds retained, times replication factor. Tripling retention triples the disk for that topic, multiplied again by every replica.

That arithmetic is why teams set aggressive short retention and then discover, during an incident, that they cannot replay far enough back. The tension is real: retention is simultaneously your replay capability and one of your larger storage line items.

Tiered storage is the structural answer where it is available. Recent segments stay on fast broker-local disk, and older segments move to cheap object storage while remaining readable through the ordinary consumer API. The effect is that long retention stops costing broker disk, which decouples “how far back can I replay” from “how much fast storage must I buy.” For deletion, audit, and compliance topics that need long windows for correctness rather than for throughput, that is exactly the right trade.

Without tiered storage, the pragmatic pattern is differential retention: short windows for high-volume operational streams, long windows for the small number of low-volume topics where missing an event is a correctness or compliance failure. Retention is a per-topic decision precisely so you can make it unevenly.

Replication factor and the environment problem

The fourth setting worth mentioning is replication factor, because it interacts with everything above and is where a specific mistake keeps happening. Replication factor determines how many brokers hold a copy of each partition, and therefore how many broker failures the topic survives. Production wants a replication factor above one. A single-broker development cluster physically cannot provide it.

The mistake is encoding the production value in the shared topic declaration, which then fails to apply on any smaller environment, or encoding the development value and quietly shipping unreplicated topics to production. The discipline that avoids both: the base registry carries the smallest-tier value, and the production value is applied through an environment overlay. The base declaration never contains the production number, so there is exactly one place where tier-specific values live and no chance of the wrong one leaking into the wrong environment. That pattern generalizes well beyond Kafka, and it is the subject of Environment Config Overlays for Kubernetes.

Compaction has a delay you should know about

One practical detail surprises people the first time they rely on compaction: it is not immediate. Kafka compacts in the background, and records are only eligible once they are in a closed segment and any configured minimum-dirty-ratio and delay thresholds are met. Until then, superseded records for a key are still present and still delivered.

The consequence is that a consumer replaying a compacted topic may see several records for the same key, not just the latest. Any consumer rebuilding state from a compacted topic must therefore apply records in order and let later ones win, rather than assuming one record per key. Treating compaction as a guarantee of uniqueness is a bug that only appears under real timing.

The related gotcha is deletion within a compacted topic. Removing a key requires publishing a tombstone, a record with that key and a null value, which compaction eventually collects along with the key’s history. Tombstones themselves are retained for a configured period so consumers that are offline can still observe the deletion, which is the same not-too-short-retention reasoning as everywhere else in this post.

A topic configuration checklist

When declaring a new topic:

• Partitions sized from required throughput and consumer parallelism, with modest headroom. Not a guess, not a large round number.
• You have accepted that partition count can grow but never shrink, and that growing it disturbs key ordering.
• Key chosen so that events needing relative order share a key (usually the aggregate ID).
• Cleanup policy chosen semantically: delete for facts, compact for state.
• Compaction is paired with infinite retention: never compaction plus a time window.
• Retention covers both your replay needs and your worst realistic consumer outage.
• Deletion, compliance, and audit-adjacent topics get deliberately longer retention.
• Replication factor comes from an environment overlay, not from the shared base declaration.
• All of the above is declared in the canonical registry and reviewed in a pull request.

What I’d do differently

The mistake I have watched most often is accepting defaults at topic-creation time and treating these as knobs to revisit later. Two of the three are not really revisitable. Partition count can go up but never down, and going up perturbs ordering. Retention can be extended, but only for data you still have, so extending it after an expiry does not recover what already aged out. The window to make these decisions cheaply is the moment the topic is created, and it never reopens.

If I were setting up an event bus again, I would make partitions, retention, and cleanup policy required fields with a written justification in the topic registry, so that no topic can be added without someone having thought about all three. The review question that catches the most future pain is the simplest one: “what happens if a consumer of this topic is down for a week?” If the answer is “it silently misses data and nobody finds out,” the retention is wrong, and that is far cheaper to learn in a pull request than in a compliance audit.

The other thing worth doing early is writing down the compaction rule explicitly, because compaction is the setting most likely to be applied for the wrong reason. It looks like a storage optimization and it is actually a semantic change to what the topic is. A team that reaches for compact to save disk on an audit stream has not saved disk; they have deleted their audit trail, and they will not notice until someone needs it.

Sources

• Apache Kafka, log compaction: kafka.apache.org/documentation/#compaction
• Apache Kafka, topic-level configuration (retention, cleanup policy): kafka.apache.org/documentation/#topicconfigs
• Apache Kafka, consumer groups and partition assignment: kafka.apache.org/documentation/#intro_consumers