The Transactional Outbox Pattern

The transactional outbox pattern exists because of one stubborn fact: your database and your message broker are two different systems, and you cannot write to both in a single transaction. So the moment a service updates a row and publishes an event about that update, there is a window where one succeeds and the other does not. The outbox closes that window by making the event part of the same database transaction as the data, then handing publication to a separate relay.

It is the least glamorous pattern in event-driven architecture and one of the highest-value. Skip it and you will eventually ship a system where the database says one thing, the event stream says another, and no amount of retry logic can reconcile them because the truth was lost at the moment of the crash.

The dual-write problem, precisely

Consider the most ordinary operation in an event-driven system. A user registers. Your identity service inserts a row into users, then publishes a user.registered event so the leaderboard can seed a row, the notification service can send a welcome, and the email service can start an onboarding sequence.

Two writes, two systems. Now enumerate the failure modes:

The database commits, then the process crashes before publishing. The user exists and nothing downstream knows. No welcome email, no leaderboard row. The user is in a broken half-state that no retry will fix, because the code that would have published is gone and nothing recorded that it still owed an event.

The event publishes, then the transaction rolls back. Now the opposite: downstream services react to a registration that never happened. The leaderboard seeds a row for a user that does not exist. Every consumer is corrupted by an event that was, in the end, a lie.

You publish first, then write, and the write fails. Same as above. You write first, then publish, and the publish fails. Same as the first case.

There is no ordering of two independent writes that is safe. This is the dual-write problem, and it is not solvable by being careful. It is solvable only by making the two writes into one write.

What is the transactional outbox pattern?

The transactional outbox pattern makes the event durable in the same transaction as the data. Instead of publishing to the broker inside your business transaction, you insert the event as a row into an outbox table in your own database. Both the business write and the outbox insert commit together, atomically. A separate relay process then reads unpublished outbox rows and pushes them to the broker.

Because the outbox insert is part of the transaction, the two disaster cases disappear:

• If the transaction commits, the event is durably recorded and will be published, because the relay will find it. Even if the service crashes immediately after commit, the row is sitting in the outbox waiting.
• If the transaction rolls back, the outbox row rolls back with it. No event is ever published for a change that did not happen.

The trade you are making is explicit: you give up publishing immediately in exchange for never being inconsistent. The event arrives at the broker slightly later, once the relay picks it up. In return, “the data changed but nobody was told” becomes structurally impossible.

How the outbox works, concretely

The write side is the simple half. Inside one transaction, you do your business write and insert the event:

BEGIN;

INSERT INTO users (id, email, created_at)
VALUES ('u_123', '[email protected]', now());

INSERT INTO outbox (id, aggregate_id, topic, event_type, payload, created_at)
VALUES (
  gen_random_uuid(),
  'u_123',
  'user.registered',
  'UserRegisteredEvent',
  '{"user_id":"u_123", ...}'::jsonb,
  now()
);

COMMIT;

That is the whole guarantee. One transaction, both facts, atomic.

The outbox row should carry everything the relay needs to publish without consulting anything else: the destination topic, the event type, the serialized payload, and a key for partitioning (usually the aggregate ID, so all events for one entity land on the same partition and stay ordered). Store the payload in whatever your broker expects, protobuf bytes, JSON, or Avro.

The read side is the relay: a loop that finds unpublished rows, publishes them, and marks them done.

loop:
  rows = SELECT * FROM outbox
         WHERE published_at IS NULL
         ORDER BY created_at
         LIMIT 500
         FOR UPDATE SKIP LOCKED;

  for row in rows:
      broker.publish(row.topic, key=row.aggregate_id, value=row.payload)
      UPDATE outbox SET published_at = now() WHERE id = row.id;

FOR UPDATE SKIP LOCKED is the detail that lets you run several relay instances safely: each grabs a disjoint batch instead of fighting over the same rows. Ordering by created_at (or a monotonic sequence) preserves the order events were produced.

Polling relay or change data capture?

There are two ways to build the relay, and the choice is a real trade rather than an obvious upgrade.

	Polling relay	Change data capture (CDC)
Mechanism	Query the outbox table on an interval	Tail the database replication log
Latency	Bounded by poll interval	Near-real-time
Load on DB	Continuous queries, even when idle	Reads the log, minimal query load
Operational complexity	Low (it’s a loop you own)	Higher (connector, offsets, schema handling)
Failure modes	Obvious and easy to reason about	Connector lag, log retention, replication slots
Good fit	Most systems, especially early	High volume, tight latency, mature ops

Polling is a background job with a SELECT. You can write it in an afternoon, debug it with a query, and it fails in ways anyone on the team understands. Its costs are a latency floor equal to your poll interval and a steady trickle of queries against the outbox table.

CDC tails the database’s own replication log (using something like Debezium) and publishes rows as they commit, which removes the polling load and gets latency close to real time. The cost is a new distributed system in your stack: a connector to run and upgrade, replication slots that can pin disk if the connector stalls, and offset management to reason about during incidents.

The honest recommendation is to start with polling. A poll interval of a second or less is imperceptible for almost every business event, and the operational simplicity is worth a great deal. Move to CDC when you have measured that polling latency or polling load is genuinely a problem, not because CDC is the more sophisticated answer. This is the same “earn the complexity” logic as deciding whether you need a service mesh.

Does the outbox guarantee exactly-once delivery?

No, and believing it does is the most common misunderstanding of the pattern. The outbox guarantees that an event is never lost and is never published for a change that rolled back. That is at-least-once delivery, not exactly-once.

The duplicate window is easy to see in the relay loop above. The relay publishes a row to the broker, then updates published_at. If it crashes between those two statements, the event is already on the broker but the row still looks unpublished. On restart, the relay publishes it again. The consumer sees the event twice.

You cannot close that gap by reordering, either: mark the row published first and a crash before the publish loses the event entirely, which is worse. Any two-system operation has this property, which is exactly the dual-write problem reappearing one layer down.

Give every outbox row a stable event ID, publish that ID with the event, and have every consumer record the IDs it has applied and skip repeats. That is the standard idempotency key discipline applied to consumption, and it is what makes the whole event-driven system safe to retry, replay, and reprocess. It is also what makes Kafka replay survivable, since a replay is nothing but deliberate duplicate delivery.

When should you not use the transactional outbox?

The outbox is not free. It adds a table, a relay to operate, latency between commit and publish, and rows to clean up. So it is worth naming the conditions under which skipping it is legitimate, rather than treating it as dogma.

The discipline I hold on the platform I run is that every producer writes through a transactional outbox, with exactly one deliberate exemption. That exemption is ephemeral presence telemetry, which is produced fire-and-forget, straight to the broker. It qualifies on two conditions that must both hold:

1. The producing service holds no database state that the event needs to be consistent with. There is no dual write here, because there is no first write. Presence is derived from a live connection, not from a committed row, so there is nothing for the event to disagree with.
2. The stream is loss-tolerant. A dropped presence heartbeat is corrected by the next one seconds later. Losing one changes nothing that anyone can observe.

If only the first condition holds, you still want durability. If only the second holds, you still have a dual write that can announce a rolled-back change. Both together, and the outbox is pure overhead: you would be paying latency and storage to protect a guarantee nobody needs.

That is the test to apply to any candidate exemption. Almost nothing passes it. Business events (registrations, payments, subscription changes, anything a user would notice) fail condition two immediately, because losing one is a support ticket at best and a lost payment at worst.

Outbox versus the alternatives

The outbox is not the only answer to the dual-write problem, and it helps to know why the alternatives lose in most situations.

Approach	How it works	Why it usually loses
Transactional outbox	Event written in the business transaction; relay publishes	Adds a table, a relay, and publish latency (the price of correctness)
Publish then write	Emit to broker, then commit DB	Announces changes that may roll back; corrupts consumers
Write then publish	Commit DB, then emit	Loses events on crash; no record that an event was owed
Two-phase commit (XA)	Distributed transaction across DB and broker	Poor performance, poor broker support, blocking on coordinator failure
Event sourcing	The event log is the source of truth	Solves it completely, but re-architects your whole persistence model
CDC without an outbox	Publish raw table changes from the replication log	Leaks your internal schema as your public event contract

Two of those deserve a longer look, because they are real designs rather than mistakes.

Event sourcing genuinely eliminates the dual write, because there is only one write: the event. State is derived by folding events rather than stored separately, so the two can never disagree. That is elegant, and it is a far larger commitment than adopting the outbox, it changes how every query works, how you handle schema evolution, and how new engineers reason about the system. The outbox gives you consistent events while keeping ordinary state-oriented persistence, which is why most teams should reach for it first.

CDC without an outbox publishes changes to your business tables directly from the replication log. It removes the outbox table entirely and it is tempting for that reason. The problem is contract coupling: your users table becomes your published event schema, so a column rename is a breaking change for every downstream consumer, and internal refactors leak outward. The outbox lets you publish a deliberate event shape that is decoupled from your storage layout, which is the whole point of a contract. If you use CDC, point it at the outbox table, not at your domain tables.

Designing the outbox table

The schema is small, and a few columns earn their place beyond the obvious ones:

Column	Purpose
id	Stable event ID; travels with the event for consumer idempotency
aggregate_id	Partition key, so per-entity ordering is preserved
topic	Destination, so the relay needs no per-event logic
event_type	Fully-qualified type name, binding payload to schema
payload	Serialized event body
created_at	Ordering, and the basis for lag measurement
published_at	Null until published; the relay’s work queue
attempts	Retry count, so a poisoned row can be detected and dead-lettered

The design principle behind it: the relay should be able to publish a row without understanding it. If the relay has to consult service code or look up which topic an event belongs on, it becomes coupled to every domain in the system. Putting the destination and type on the row keeps the relay a dumb, reliable pipe, which is exactly what you want from the component whose failure silently stops all events.

Operating an outbox in production

The pattern is simple; running it well has a few sharp edges worth knowing before you meet them.

The table grows forever unless you prune it. Every event ever produced leaves a row. Delete or archive published rows on a schedule, and keep a window long enough to debug from (a few days is usually plenty). An unpruned outbox eventually makes the relay’s own query slow, which is a self-inflicted latency problem.

Index for the relay’s query, not for reads. The relay repeatedly asks for the oldest unpublished rows. A partial index on unpublished rows keeps that query fast even as the table accumulates history, because the index only contains the small working set rather than every event you have ever emitted.

Monitor outbox lag, and alert on it. The number that matters is the age of the oldest unpublished row. If that climbs, your relay is down or falling behind, and events are silently not reaching consumers even though every transaction is committing fine. This is a genuinely dangerous failure because the application looks perfectly healthy from the outside: writes succeed, no errors, and the rest of the system quietly starves. It belongs on the operator dashboard and it should page on a real backlog, exactly the way you would alert on consumer lag.

Preserve ordering where ordering matters. Publishing keyed by aggregate ID keeps all events for one entity on one partition and therefore in order. Across different entities, order is not guaranteed and should not be relied on. If your consumers need a global order, that is a much heavier requirement and usually a design smell worth revisiting.

Decide what happens to a permanently unpublishable row. If the broker rejects an event forever (an oversized payload, a topic that no longer exists), the relay will retry it in a loop and block everything behind it. Give the relay a failure threshold and a dead-letter path so one poisoned row cannot halt the entire stream.

A transactional outbox checklist

Before you rely on an outbox in production:

• The business write and the outbox insert are in the same transaction, always.
• The outbox row carries everything needed to publish: topic, event type, payload, and a partition key.
• Every event has a stable event ID that travels with it.
• Every consumer is idempotent by that ID. This is not optional.
• The relay uses SKIP LOCKED (or equivalent) so multiple instances are safe.
• Published rows are pruned on a schedule; a partial index keeps the relay query fast.
• Outbox lag is monitored and alerted on: oldest unpublished row age.
• A poisoned row has a failure threshold and a dead-letter path.
• Any exemption from the outbox is documented and passes both conditions: no DB state, loss-tolerant.

What I’d do differently

The mistake I would warn against is treating the outbox as a thing you add later, once you have seen inconsistency in production. By then you have consumers built on the assumption that events are reliable and a backlog of quiet corruption nobody has noticed yet, because the failure is invisible: a missing event does not throw, it just means something downstream never happened. Nobody files a bug for an email that was never sent.

If I were starting an event-driven system again, I would put the outbox in the very first service that publishes anything, and I would write “every consumer is idempotent by event ID” into the platform rules before the second service existed. Those two together are what make everything downstream tractable, replay, reprocessing, adding a new consumer that needs history, and retrofitting them across a fleet of services is dramatically more expensive than starting with them.

The other thing I would do earlier is write down the exemption test rather than leaving it to judgment. Without a stated rule, every team eventually argues that their event is the special case that does not need durability, and the pattern erodes one exception at a time. Two conditions, both required, documented once: that is what keeps a discipline intact when you are not in the room.

Sources

• Microservices.io, Pattern: Transactional outbox: microservices.io/patterns/data/transactional-outbox.html
• Debezium, Reliable microservices data exchange with the outbox pattern: debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern
• Apache Kafka, delivery semantics: kafka.apache.org/documentation/#semantics