GDPR Deletion in Event-Driven Systems

Deleting a user from a microservice system is a fan-out problem, because each service owns its own data and there is no single database to delete from. That part is well understood. The part that quietly ruins implementations is a semantics question almost nobody asks up front: what does your deletion event actually mean? If it means “a user requested deletion,” every consumer that purges on receipt has just performed an irreversible act on a request that might be canceled or might fail. If it means “erasure has been executed,” purging on receipt is always safe, because receipt is proof.

Getting that distinction right is the difference between a deletion pipeline you can trust and one that occasionally destroys the data of users who changed their minds.

This is an engineering discussion of building deletion pipelines, not legal advice. Talk to counsel about what your obligations actually are.

Why deletion is hard in a distributed system

In a monolith with one database, erasure is a transaction. In a system where each service owns its own data (which is the whole point of the boundary) a user’s data is scattered across many stores that no single component can reach. The identity service has the account. The leaderboard has denormalized rows. Search has an index. Notifications has preferences and delivery history. Analytics has events.

There is no central place to delete from, and building one would mean giving something direct access to every service’s database, which destroys the boundary you designed. So erasure becomes a coordination problem: announce it once, and have every owner purge what it holds.

That is exactly the shape an event bus is for, which is why deletion fan-out is usually implemented as an event. The trouble is that deletion is unlike every other event on your bus in one crucial respect: it is irreversible. Every other consumer action can be retried, corrected, or recomputed. A purge cannot be undone. That asymmetry is what makes the semantics matter so much.

Should a deletion event mean “requested” or “executed”?

Executed. This is the single most important decision in the design, and the intuitive choice is the wrong one.

The intuitive design goes: user clicks delete, service publishes user.deletion.requested, every consumer purges. It is clean, it is immediate, and it is dangerous, because the event has announced an intention while consumers have taken an irreversible action on it. Now consider what happens when reality intrudes:

The user changes their mind. Most products offer a grace period precisely because account deletion is a decision people regret. But the fan-out already happened. Their leaderboard history, their messages, their preferences are gone across a dozen services. You can restore the account row and nothing else. The cancellation is a lie.

The upstream deletion fails. The event went out, consumers purged, and then the primary erasure hit a constraint violation or an outage and rolled back. Now the account still exists, but every other service has destroyed its data for that user. The system is in a state with no name and no recovery path.

The request was fraudulent or mistaken. An account takeover triggers a deletion, or a support agent fat-fingers an ID. With request-semantics, the damage is instant and total across every service.

Now flip the semantics. The event means: this erasure has already been performed, irreversibly, upstream. Every one of those cases evaporates. There is nothing to cancel because the event is only emitted after the cancellation window closed. There is no failed-upstream case because the event is emitted after the upstream delete succeeded. Consumers can purge on receipt with total confidence, precisely because receipt proves execution.

How do you handle a grace period?

Keep it entirely in your own state, off the event bus. The discipline I hold on the platform I run is that a deletion request lives in job rows inside the owning service for the whole grace window, and nothing irreversible fans out during that window. The erasure event is emitted only at the end, by the worker that actually performs the deletion, after it has succeeded.

The full lifecycle looks like this:

1. User requests deletion. The identity service writes a job row: this user, requested at this time, scheduled for erasure after the grace period. No fan-out event. Optionally an event that consumers treat as informational only, never as a trigger to purge.
2. The grace period runs. The request is fully cancelable. The user can log back in and change their mind, and nothing has been destroyed anywhere, because nothing irreversible has been announced.
3. The user cancels (maybe). The job row is canceled. If you emitted an informational event at request time, you emit a corresponding cancellation event now so any non-destructive state (a “pending deletion” banner, a suppressed email) is reverted.
4. The grace expires. A scheduled erasure worker picks up the job, performs the irreversible deletion of the primary data, and confirms it succeeded.
5. Only now, the worker emits the erasure event. Jobs that were canceled inside the grace are skipped entirely, they never reach this step, so they never fan out.
6. Every consumer purges on receipt. Safe by construction.

The property that makes this whole design hold together is the ordering in step 5: the event is emitted by the worker that did the erasure, after the erasure, only for jobs that actually executed. That is what makes “purge on receipt” a correct rule rather than a hopeful one.

Note also who emits it. It is not the API handler that received the user’s click, and not a scheduler that merely knows the deadline passed. It is the component that performed the irreversible act and observed it succeed. Emitting from anywhere else reintroduces the gap between intention and fact.

Request-time state belongs in the database, not the bus

A useful way to hold the whole design: the bus carries facts; the database carries intentions.

A deletion request is an intention. It is mutable (cancelable), it is scoped to one service, and it has no business being a trigger for irreversible action elsewhere. It lives in job rows, where it can be queried, canceled, retried, and audited by the service that owns it.

An executed erasure is a fact. It is immutable, it is globally relevant, and it is exactly what other services need to know. That is what goes on the bus.

Once you draw that line, several other questions answer themselves. What if the user cancels? The intention is deleted from the job table; no fact was ever published. What if the erasure worker crashes mid-job? The job row is still there, still pending; it retries, and only publishes when it truly finishes. What if you need to show the user a “deletion scheduled for X” banner? That is a query against the job row, not a subscription to an event.

Making the fan-out itself reliable

The semantics get you a safe event. Delivering it reliably is the ordinary distributed-systems work, and every piece of it matters more than usual because a missed deletion is a compliance failure that produces no error.

Publish through the outbox. The erasure and its event must be atomic. If the worker deletes the data and crashes before publishing, no consumer ever learns, and the user’s data survives everywhere except the primary store, silently, forever. Writing the event to the transactional outbox in the same transaction as the erasure closes that gap.

Make consumers idempotent. The event will be delivered more than once, because at-least-once delivery is what you have. A purge is naturally idempotent in the happy case (deleting already-deleted rows is a no-op), but the surrounding work often is not, so key on the event ID like any other idempotent consumer.

Retain the event far longer than ordinary events. This is the failure mode with no alarm. If a consumer is down for longer than the topic’s retention and the deletion event expires while it is offline, that service comes back, resumes from the earliest available offset, and never sees the deletion. It holds the user’s data indefinitely. Nothing errors, nothing alerts, and no dashboard shows it. Deletion topics should carry retention sized so that no realistic outage can cause a miss, the reasoning covered in Kafka partitions, retention, and compaction.

Never compact a deletion topic. Compaction retains the latest record per key and discards the rest. For a deletion stream keyed by user, that is either useless or actively harmful, and it destroys the audit trail of what was erased when.

Track completion, do not assume it. The bus tells you the event was published, not that eleven services purged. If you need to demonstrate erasure completeness, have consumers acknowledge, and track outstanding acknowledgements per deletion. Without that, “we published the event” is the strongest claim you can make, and it is weaker than it sounds.

Backups, derived data, and the parts people forget

Two categories of data routinely escape a fan-out that is otherwise correct.

Derived and denormalized copies. Search indexes, caches, materialized views, analytics warehouses, and read models all hold copies of user data that the fan-out must reach. Each of these is a consumer that needs to purge, and they are easy to forget precisely because they are derived rather than authoritative. The registry practice from topic governance helps here: if every topic declares its known consumers, the deletion topic’s consumer list is a checklist of everything that must act.

Backups and the event log itself. Backups contain the data you just erased, and the event log may contain payloads with personal data in them. These are genuine problems with no clean event-driven answer, and the common engineering approaches are to bound backup retention so copies age out on a defined schedule, to keep personal data out of event payloads (carry an ID and let consumers look up what they need), or to use crypto-shredding, where per-user encryption keys are destroyed so remaining ciphertext is unrecoverable. Which of these you need is a question for counsel; the engineering point is to decide deliberately rather than discover the gap during an audit.

The design principle worth internalizing from this: the less personal data you put on the bus, the smaller your deletion problem is. An event carrying a user ID and nothing else is trivial to live with. An event carrying a full profile snapshot means every topic retaining that event is now a copy of personal data you must reason about.

Data export is the same fan-out, running backwards

The right to obtain a copy of your data is the mirror image of erasure, and it is worth designing at the same time because it exercises the identical machinery: every service holds a piece, and something must collect from all of them.

The semantics are gentler, because export is not irreversible. A duplicate export is harmless; a failed export can simply be retried. That means you can use ordinary request/response gathering rather than the strict emission contract erasure requires.

What export does share with deletion is the completeness problem. Both need an accurate answer to “which services hold data for this user?”, and both fail silently when a service is forgotten. An export that misses a service produces an incomplete response nobody notices; a deletion that misses one leaves data behind. That shared dependency is a strong argument for maintaining an explicit, reviewed inventory of which services hold personal data, and having both pipelines consume that same list. When a new service starts storing user data, it should land on one list and be picked up by both flows.

The practical shape most teams land on: a coordinator asks each registered service for its data for a subject, assembles the responses, and produces the artifact, with per-service timeouts and a clear record of any service that failed to respond. That last part matters, an export that quietly omits a service that timed out is worse than one that reports the gap.

How do you test a deletion pipeline?

Deletion is uniquely hard to test because the failure is silent and the action is irreversible, which is exactly the combination that keeps people from testing it properly. It is also the reason you must.

Test the fan-out completeness, not just the primary delete. The interesting assertion is not “the user row is gone” but “no service holds data for this subject.” That means a test that creates a user, exercises enough of the product to leave traces in several services, runs the erasure, and then checks every service for residue. This is an integration test by necessity; a unit test of the identity service proves almost nothing about the property you care about.

Test the grace-period cancellation explicitly. Request deletion, cancel inside the window, then assert that nothing was purged anywhere. This is the test that catches the request-versus-executed semantics bug directly, and it is the one most likely to be missing.

Test the delayed-consumer case. Take a consumer offline, run a deletion, bring the consumer back after a delay, and assert it still purges. This is the test that would catch a retention window shorter than your realistic outage, the silent compliance gap that nothing else surfaces.

Test idempotency. Deliver the deletion event twice and assert the second delivery is harmless. At-least-once delivery guarantees this will happen in production eventually.

The uncomfortable part is that a genuinely complete test requires a way to enumerate every store that could hold subject data, which is the same inventory the export flow needs. Teams that maintain that inventory can test deletion meaningfully. Teams that do not are asserting a property they have no way to check, and the gap between “we published the event” and “the data is gone everywhere” is precisely where compliance findings live.

A deletion fan-out checklist

Before you trust an erasure pipeline:

• The deletion event means erasure executed, never requested.
• It is emitted only by the worker that performed the erasure, after success.
• Grace-period state lives in job rows, not on the bus; canceled jobs never fan out.
• A separate cancellation event reverts any non-destructive request-time state.
• The event is published through the transactional outbox, atomic with the erasure.
• Consumers are idempotent and purge on receipt.
• Retention exceeds any realistic consumer outage; the topic is never compacted.
• Derived stores (search, cache, read models, analytics) are enumerated as consumers.
• Backups and event payloads have a deliberate policy (bounded retention, ID-only payloads, or crypto-shredding).
• If you must prove completeness, consumers acknowledge and outstanding acks are tracked.

Soft delete is not deletion

A pattern worth naming because it is so common: marking a row deleted_at = now() and filtering it out of queries. Soft delete is a useful product mechanism, and it is not erasure. The data is still there, still in backups, still visible to anyone with database access, and still returned by any query that forgets the filter.

The two serve different purposes and should not be confused. Soft delete supports the cancelable grace period, it makes an account disappear from the product immediately while remaining recoverable, which is exactly what you want during the window. Hard erasure is what the deletion event announces, and it happens at the end.

The failure mode is a pipeline that soft-deletes everywhere and considers itself done. From the outside it looks correct: the user is gone from the UI, every service filters them out, nothing references them. The data is fully intact. That is a compliance gap that presents as a working feature, which is the hardest kind to catch, and it is why the completeness test above must assert on actual absence rather than on the product’s behavior.

What I’d do differently

The mistake I would flag hardest is treating deletion as just another event, because it looks like one. It has a topic, a payload, and consumers, so it gets designed with the same care as user.updated. But it is the only event on your bus where a consumer’s normal response (react to what you just learned) is irreversible, and that single property should change how you design it.

If I were building this again, I would decide the emission semantics before writing any code, and I would write them into the topic’s description so nobody has to infer them. “This event means the erasure has already been executed; purge on receipt is safe” is one sentence, and it is the difference between a consumer author doing the right thing by default and doing something destructive by reasonable-looking guess.

The second thing I would do earlier is keep personal data off the bus entirely. Carrying only an identifier and letting each consumer look up what it needs makes the whole erasure story dramatically simpler, because the event log stops being another place the data lives. That decision is nearly free at the start of a system and quite expensive once sixty topics carry rich payloads.

Sources

• Microservices.io, Pattern: Transactional outbox: microservices.io/patterns/data/transactional-outbox.html
• Apache Kafka, log compaction and retention: kafka.apache.org/documentation/#compaction
• European Commission, right to erasure overview: commission.europa.eu/law/law-topic/data-protection_en