Kafka Topic Naming and Governance

An event bus with a dozen topics governs itself. At fifty or sixty, without a Kafka topic naming convention and a single source of truth for what exists, it becomes an archaeology project: nobody can tell which topics are live, who owns them, what shape the payload is, or whether the one called user-events-v2-new is safe to delete. The fix is unglamorous and cheap: a strict naming law, and one canonical registry in version control that is the only way a topic comes into existence.

This is governance in the boring, useful sense. It costs almost nothing to establish on day one and is painful to retrofit once forty topics have been created by hand across three environments that no longer match.

Why topic sprawl is a real failure mode

Topics are trivially easy to create. Someone needs a new stream, runs a command against the cluster or lets auto-creation handle it, and the topic exists. That convenience is exactly the problem, because nothing about that act is recorded, reviewed, or reproducible.

A few months of that and you have the symptoms every team recognizes. Three topics that look like they do the same thing, and no one is sure which is current. A topic nobody will delete because nobody can prove it has no consumers. Staging and production with different topic sets, so a deploy that works in one fails in the other. A payload shape that changed six months ago with no record of who changed it or why.

None of this is a Kafka problem. It is a governance problem, and it shows up in exactly the same way that undeclared service dependencies produce dependency cycles: individually reasonable local decisions accumulating into a system nobody can reason about globally. This post is part of the Distributed systems patterns series.

What is a good Kafka topic naming convention?

A good convention encodes three things in a fixed order: the domain that owns it, the entity it concerns, and what happened to that entity, in past tense. Lowercase, dotted, no abbreviations that only one team understands. The structure I hold to is:

<domain>.<entity>.<event-past-tense>

Which produces names that are self-describing at a glance:

user.registered
user.deleted
race.started
race.finished
payment.captured
subscription.changed
leaderboard.updated

You can tell, without opening any documentation, which part of the system owns each topic, what it is about, and what kind of thing it announces. That legibility is the entire deliverable. When someone lands on a topic name in a consumer config at 3 a.m., the name should answer their first question before they have to go looking.

The specific scheme matters less than you would expect. Some teams use underscores, some prefix with an environment, some add the version to the topic name. Those are defensible variations. What is not defensible is inconsistency, because the value of a convention is that you can predict a name you have never seen, and one exception destroys that property for the whole set.

Should Kafka topic names use past tense?

Yes, for event topics, and it is a more load-bearing rule than it looks. An event is a record of something that already happened. user.registered is a fact: at some point, a user registered, and here is the evidence. register.user reads as an instruction: someone should go register a user. Those are different architectures.

The naming pushes teams toward the right one. When topics are named in past tense, producers naturally publish facts and consumers naturally react to them, which is what an event bus is for. When topics are named as commands, teams drift into using Kafka as a request/response channel with implied ownership of downstream behavior, and you get the tight coupling that event-driven architecture was supposed to remove.

The exception worth acknowledging: genuine command topics exist in some designs (job queues, work dispatch). Those are legitimately imperative and should be named as such, and kept visibly separate from your event topics so the distinction stays clear.

How do you manage Kafka topics across many services?

Keep one canonical registry file in version control that declares every topic and its full configuration, and apply that file to the cluster with a script. Topics are never created by hand on a live cluster, and auto-creation is disabled. The registry is the single source of truth; the cluster is a projection of it.

This is the practice I hold on the platform I run, where the bus runs to roughly sixty topics. A single registry file declares each one, and a script reads that file and applies it. The properties that fall out of it are worth enumerating, because each one solves a real problem from the sprawl list above:

• Adding a topic is a pull request. It gets reviewed like code. Someone asks whether the name fits the convention, whether the retention is right, whether it duplicates an existing topic. That review is the governance.
• The topic list is diffable and has history. You can see when a topic appeared, who added it, and what the config was before someone changed it.
• Environments are reproducible. A new cluster is bootstrapped by running the script. Staging and production cannot drift, because they are generated from the same declaration.
• Nothing exists by accident. If a topic is on the cluster and not in the registry, that is a detectable anomaly rather than a mystery.

Each registry entry should carry more than the name. At minimum: the partition count, replication factor, retention, cleanup policy, the key it is partitioned by, the fully-qualified event type it carries, the owning service, and a one-line description naming the known consumers. That last field is the one people skip and later wish they had, because “who consumes this?” is the question that blocks every deletion.

Who should own a Kafka topic?

Exactly one producing service owns each topic, and that ownership is declared explicitly in the registry. The owner is accountable for the topic’s schema, its configuration, and any change to either. Many services may consume it; only one may produce to it and only one is responsible for its contract.

Single ownership is what makes the contract enforceable. If two services produce to the same topic, they must agree on the payload shape forever, coordinate every change, and neither is clearly on the hook when a consumer breaks. That is the same shared-ownership failure that dissolves service boundaries, just expressed through a topic instead of a database.

Declaring the owner in the registry converts a social assumption into a documented fact. When a consumer breaks because a payload changed, there is no debate about whose change it was or who fixes it. And when someone proposes producing to an existing topic from a second service, the registry makes that proposal visible in review, where the right answer is usually “publish your own event to your own topic instead.”

Bind the topic to a versioned event type

A topic name says what stream this is. It does not say what shape the payload has. The registry should bind the two, declaring the fully-qualified event type each topic carries:

topic:      user.registered
event type: identity.v1.UserRegisteredEvent
owner:      identity-service

The parallel naming is deliberate. <domain>.v1.<Entity><PastTenseVerb>Event mirrors the topic’s <domain>.<entity>.<past-tense> so the two read as the same fact expressed in two registries: one for the transport, one for the schema. When someone reads the topic name they can derive the message type, and vice versa.

The v1 in the type is where schema evolution lives. Compatible changes happen inside v1 under the ordinary Protobuf evolution rules: add fields with new numbers, reserve what you remove, never retype or recycle a field number. A genuinely incompatible redesign becomes v2 as a new type, and the migration is explicit rather than accidental. Because the version is structural rather than encoded in the topic name, you avoid the trap of topics named user-events-v2 that nobody can retire because the version means something different to each consumer.

Should the version go in the topic name?

Usually not. Putting a version in the topic name (user.registered.v2) is a common instinct and it creates more problems than it solves, because it conflates two different kinds of change.

Most schema changes are compatible (adding a field, deprecating one) and require no new topic at all. Consumers that do not know the new field ignore it. If you mint a new topic for every compatible change, you accumulate v2, v3, v4 topics that all carry the same logical stream, and every consumer must be migrated for changes that did not require migration.

Genuinely incompatible changes are rare, and when they happen the version belongs on the event type, not the topic: identity.v2.UserRegisteredEvent. That way the topic remains the stable name of the stream (the thing consumers subscribe to and dashboards reference) while the payload contract carries its own version. You can even run both types on one topic during a migration window, with consumers handling whichever they understand.

There is one legitimate exception: a rewrite so total that the old and new streams should not be mixed at all, with different keys, different semantics, and a long dual-write period. Then a new topic is honest, because it really is a different stream. That is rare enough that it should feel like an event, not a routine.

How do you retire a Kafka topic safely?

Retiring is where governance pays for itself, because the blocking question is always the same: who still consumes this? Without a registry, answering it means grepping every repository and hoping nothing dynamic constructs the name at runtime. With a registry that records consumers, it is a lookup.

The sequence that avoids breaking someone:

1. Mark it deprecated in the registry with a date and a replacement. The declaration is now the announcement.
2. Verify no active consumer groups. The registry tells you who should be consuming; the cluster tells you who is. Check both, because they disagree exactly when it matters.
3. Stop producing. The topic goes quiet but still exists, so a forgotten consumer fails loudly on missing data rather than silently on a missing topic.
4. Wait out the retention window. Anything still needed is still readable during this period.
5. Delete, via the registry. Remove the entry, run the apply script. The deletion is a reviewed commit with history.

Step 3 is the one people skip, and it is the cheapest insurance in the sequence: a period where the topic exists but is empty gives any straggler a clear, debuggable failure instead of a confusing one.

Environment prefixes and shared clusters

If several environments share one Kafka cluster, topic names need an environment prefix (staging.user.registered) so streams cannot collide. That prefix should be applied by the apply script from an environment overlay, never written into the canonical registry.

The reason is the same single-source principle that governs the rest of the file: the registry describes the logical topics your system has, once. Environment placement is a deployment concern. If you hardcode prefixes into the registry, you end up with one entry per environment per topic, which triples the file and reintroduces exactly the drift that a single source of truth was meant to eliminate. This is the same base-plus-overlay discipline covered in Environment Config Overlays for Kubernetes.

The stronger option, where you can afford it, is a separate cluster per environment and no prefixes at all. Names stay clean, isolation is physical rather than by convention, and a misconfigured consumer in staging cannot read production data, a failure mode that prefixes prevent only as long as everyone gets the prefix right.

A worked example of the whole discipline

Here is what one entry looks like when the conventions are followed, in the shape a registry entry actually takes:

Field	Value	Why it’s there
name	user.registered	Convention-compliant, self-describing
partitions	sized to the tier	Throughput and consumer parallelism
key	user_id	All events for a user stay ordered together
event type	identity.v1.UserRegisteredEvent	Binds transport to schema
owner	identity-service	One accountable producer
cleanup policy	delete with explicit retention	Fact stream, not state
description	consumers: leaderboard, notification, email	Answers “who breaks if this changes?”

Six fields and a sentence. That is the entire cost of governance per topic, and it removes the archaeology problem permanently. The retention and cleanup-policy choices in that table are their own decision with real consequences, covered in Kafka Partitions, Retention, and Compaction.

Name consumer groups too

Topic naming gets all the attention, and consumer group IDs quietly deserve the same discipline. A group ID determines offset ownership, so a careless one causes real damage: two unrelated services sharing a group ID will split partitions between them, each silently processing a fraction of the stream while both appear healthy.

Use a convention that encodes the consuming service and its purpose, such as <service>.<purpose>. notification-service.welcome-emails says who is consuming and why, which makes the consumer-group list on a cluster as legible as the topic list. A group named consumer-1 tells an on-call engineer nothing at the moment they most need to know whose lag is climbing.

Two rules that prevent the worst outcomes. Never share a group ID across services, if two services both need every event, they need two groups, because a shared group means each sees only part of the stream. And treat a group ID as permanent: changing it resets consumption position, so a rename is a replay, with all the duplicate-delivery consequences that implies. What looks like a cosmetic rename is a data-processing event.

Record the expected consumer groups next to each topic in the registry. That turns “which groups should exist?” into a lookup, and makes an unexpected group on the cluster a visible anomaly rather than something nobody notices until it has been quietly stealing partitions for a month.

A topic governance checklist

For an event bus that stays legible past fifty topics:

• A written naming convention: domain.entity.event-past-tense, lowercase, dotted.
• Event topics are past tense; any genuine command topics are named and grouped separately.
• One canonical registry file in version control declares every topic.
• A script applies the registry to the cluster; auto-creation is disabled.
• Every entry declares: partitions, replication factor, retention, cleanup policy, key, event type, owner, and known consumers.
• Exactly one owning service per topic.
• The event type is versioned (domain.v1.…) and evolves under compatibility rules.
• Adding or changing a topic goes through pull-request review.
• Cluster state is periodically reconciled against the registry to catch drift.

Where a schema registry fits

The topic registry says what streams exist; a schema registry enforces what their payloads may become. They are complementary, and teams often adopt one and assume it covers the other.

A schema registry stores the schema for each topic’s messages and checks new versions against a compatibility rule before allowing them. Set to backward compatibility, it will reject a change that would break existing consumers, at the moment the producer tries to register it, rather than at runtime in production. That is a genuinely valuable gate: it turns “we hope nobody ships a breaking change” into a mechanical check.

What it does not do is tell you which topics exist, who owns them, what retention they carry, or who consumes them. Those are governance facts that live in your topic registry and get reviewed in a pull request. A schema registry protects the payload contract; the topic registry protects everything around it.

If you use Protobuf, much of the compatibility discipline is already available through schema evolution rules plus a build-time breaking-change check, which many teams find sufficient without running a separate registry service. Either way, the principle holds: automate the contract check rather than relying on review to catch a field-number reuse that a human reader will almost certainly miss.

What I’d do differently

The mistake is thinking governance is something you add when the mess gets bad enough to hurt. By then you have topics whose consumers are unknown, so you cannot safely delete them; environments that have drifted, so you cannot trust staging; and payloads that changed without record, so you cannot tell whether a consumer bug is a bug or an undocumented contract change. Every one of those is cheap to prevent and expensive to reverse.

If I were standing up an event bus again, I would write the naming convention and create the registry file before the first topic existed, and disable auto-creation on day one so there is no path around it. The convention takes ten minutes to decide and the registry is a file. What they buy is that at sixty topics you can still answer, instantly and from version control, what exists, who owns it, what it carries, and who would break if it changed. That is not a small thing at 3 a.m.

The related discipline is to make deletion possible. Recording known consumers in the description is the field that seems like paperwork right up until the day you want to remove a topic and the only alternative is grepping every repository and hoping.

Sources

• Apache Kafka, topic configuration and administration: kafka.apache.org/documentation/#configuration
• Confluent, event-driven design and naming guidance: developer.confluent.io/patterns
• Protocol Buffers, updating a message type: protobuf.dev/programming-guides/proto3/#updating