The Outbox Pattern: Reliable Messaging in System Design

Saving your data and telling the rest of the system about it sounds like one step. It is two, and the gap between them is where things break. You save an order to your database, then publish an OrderCreated event so inventory, email, and shipping can react. Two writes, two systems, and nothing holding them together. It is one of the most common reliability bugs in microservices, and a classic system design interview question.

Two writes, no shared transaction

When an order is placed, your service has to do two things, and they live in two different systems with no transaction spanning both: write the order to Postgres, and publish OrderCreated to Kafka.

Whatever order you run them in, a crash in the gap leaves you broken in one of two ways.

• If the database commits and the publish fails, the event is lost. Inventory, email, and shipping never hear about the order.
• If you publish first and the database rolls back, you have a phantom event for an order that does not exist, and everything downstream acts on a lie.

Picture the first failure in production: a customer checks out, the order row commits, and right then the broker is mid-deploy, so the publish call throws. The order sits in your database, fully paid, while the warehouse, email, and analytics never learn it exists. No error reaches the user, and you find out days later from a support ticket.

A word on the obvious "fix." Reaching for a distributed transaction (2PC / XA) across the database and the broker does not save you. Kafka does not really support it, the blocking coordinator wrecks availability, and you have now coupled your database to your broker. Don't.

Make it one write

The fix is to stop writing to two systems in the request path. Insert the event into an outbox table in the same local transaction as the business change. One ACID transaction, one database: both rows commit, or neither does. This is the transactional outbox pattern.

A separate relay process then reads the outbox and delivers events to the broker asynchronously. A cross-system atomicity problem becomes one database transaction plus reliable async delivery.

One transaction, two rows

The write is a single transaction that touches two tables:

BEGIN;

-- business state change
INSERT INTO orders (id, customer_id, total_cents, status)
VALUES ('ord_9F2', 'cus_31', 4999, 'PLACED');

-- the event: same transaction, same connection
INSERT INTO outbox (id, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), 'ord_9F2', 'OrderCreated',
        '{"orderId":"ord_9F2","totalCents":4999}');

COMMIT;

The outbox table itself is small: id, aggregate_id, event_type, payload (jsonb), created_at, and published_at. Once this commits, the order and its event are atomic. No interleaving, crash, or rollback can leave one without the other.

Getting the outbox to the broker

The write is safe. Now a relay moves unpublished rows to the broker, reliably and in order. There are two ways to build it, and choosing between them is where the interview gets interesting.

Approach A: the polling publisher

A worker queries the table on an interval, publishes new rows, and marks them done. The detail that matters is claiming rows so two workers never grab the same batch:

-- claim a batch without two workers grabbing the same rows
SELECT * FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- then: publish each row, then UPDATE published_at = now()

What it wins you:

• Dead simple. No extra infrastructure.
• Works on any database, in any language.
• Full control over retries and batching.

What it costs you:

• Your latency floor is the poll interval. Tighter polling means more load.
• Constant queries and write amplification.
• You build cleanup and ordering yourself.

Approach B: change data capture (CDC)

No application code publishes anything. A connector tails the database transaction log, the Postgres WAL or MySQL binlog, and routes the committed outbox insert straight to a topic.

What it wins you:

• Near real-time, with no polling lag.
• Almost zero load on the database. It is just a log read.
• Preserves commit order and scales to high throughput.

What it costs you:

• Heavy infrastructure: Debezium, Kafka Connect, schema management.
• Coupled to the database log internals and version.
• A lagging replication slot can balloon your WAL.

Polling vs CDC

Start with polling. Graduate to CDC when latency or throughput, not résumé hype, actually demands it.

The tradeoffs that get you the offer

Knowing the pattern is table stakes. These are the follow-ups that separate a senior answer in a system design interview.

At-least-once, not exactly-once

Both relay styles can publish a row and then crash before marking it done, which means redelivery. The outbox guarantees delivery, never uniqueness. Say this out loud before the interviewer makes you.

Make consumers idempotent

Ship a stable event_id on every event and have consumers dedupe, either with an upsert or a processed-ids table. At-least-once delivery plus idempotent consumers gives you effectively-once processing, which is the honest version of "exactly-once."

-- Consumer dedupe: the insert itself is the idempotency gate.
INSERT INTO processed_events (event_id, handled_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;
-- 0 rows affected means this event was already handled, so skip it.

Ordering is not free

Key by aggregate_id so all events for one entity land on a single partition and stay ordered per entity. SKIP LOCKED across many polling workers breaks strict order; CDC preserves log order. Global ordering across everything is a myth you pay dearly for, so do not promise it.

Reading the pattern is not the same as defending it on a whiteboard. If you want to pressure-test it, wire the outbox into a real payment system design with idempotency keys, ordering, and exactly-once semantics.