Here is a line of code that looks completely innocent and is quietly lying to you:
await _db.SaveChangesAsync();
await _bus.PublishAsync(new OrderPlaced(order.Id));Save the order, tell the world. Two lines, one intent. The problem is that it isn’t one step — it’s two, against two different systems, with no shared transaction between them. And the small gap in the middle is exactly where a process can die, a network can blink, or a broker can refuse the message. When it does, your database and everyone downstream disagree about what just happened. This is the dual-write problem, and I flagged it in passing when itemising the distributed systems tax; it’s worth its own post because almost every integration eventually trips over it.
Two ways to lie to yourself
There are only two orders you can write those two lines in, and both are broken.
Database first, then publish. You commit the order, then the process crashes — or the broker is down, or the publish times out — before the message goes out. The order exists; nobody was told. A payment that never gets taken, an email that never sends, a downstream system that never hears. Silent, and usually discovered days later by a confused human.
Publish first, then save. You emit OrderPlaced, then the database commit fails on a
constraint or a deadlock. Now there’s a message racing through your system for an order that
doesn’t exist. Downstream consumers dutifully act on a phantom. This one is often worse,
because it’s a lie that looks like a success.
There’s no third ordering that saves you. The two writes are not atomic, and no amount of careful sequencing makes them atomic.
The tempting non-answer: a distributed transaction
The instinct is to reach for a distributed transaction — two-phase commit across the database and the broker, so they succeed or fail together. Resist it. 2PC is a genuine option on paper and a trap in practice: it needs a coordinator, it holds locks while it waits for the slowest participant, it blocks awkwardly when that coordinator itself fails, and plenty of modern brokers don’t support it at all. You’d be buying atomicity with availability and throughput — usually a bad trade, and one you can’t easily undo. Like most architecture calls, whether it’s worth it is it depends, but for the ordinary case of “save a row and publish an event,” it almost never is.
The better move is to stop trying to make two systems agree simultaneously, and instead make one of them the single source of truth.
The outbox pattern
The outbox pattern is almost embarrassingly simple once you see it: write the message into the same database, in the same transaction, as the business data it describes.
You add an outbox table. When you place the order, you insert the order row and a row
representing the OrderPlaced message, and you commit them together in one local transaction.
That commit is atomic — it’s a single database, the thing databases have been good at since the
1980s. Either both rows are there or neither is. The dual-write problem is gone, because there’s
no longer a dual write: there’s one write, to one system.
Then a separate relay (sometimes called a message relay or dispatcher) does the second half asynchronously: it reads unsent rows from the outbox, publishes them to the broker, and marks them sent.
[ business txn ] [ relay, out of band ]
INSERT order ┐
INSERT outbox row ├── COMMIT poll outbox → publish → mark sent
┘The atomic commit is your source of truth. The relay’s only job is to make the broker eventually catch up with what the database already durably decided.
What you’ve actually bought — and what it costs
You haven’t conjured atomicity for free; you’ve moved the hard part somewhere you can control it. The bill:
- The relay is now critical infrastructure. If it stops, messages stop flowing — they’re safe in the outbox, but they’re not going anywhere until it comes back. It needs monitoring, alerting, and a story for running exactly one publisher (or a safely-partitioned set) so you don’t get a stampede.
- You get at-least-once, not exactly-once. The relay can publish a message, then crash before it records the row as sent — so on restart it publishes it again. That’s not a bug you can design away; it’s the fundamental shape of the problem. Which means your consumers must be idempotent. (That’s a whole post of its own — and it’s the next one — because “exactly once” is a lie you’ll be tempted to believe.)
- The outbox table grows. Sent rows need pruning, or the table becomes a liability. A cleanup job or a partitioned/TTL’d table is not optional.
- Ordering is not guaranteed for free. If order matters between messages, the relay has to preserve it deliberately (sequence numbers, per-key ordering) — the broker won’t do it for you just because you inserted in order.
Polling vs tailing the log
Two ways to build the relay, and it’s a real decision:
- Poll the table. A loop that queries for unsent rows on an interval. Dead simple, works with any database, easy to reason about. The cost is latency (you publish on the next tick) and load (you’re querying constantly). For most systems this is entirely good enough — don’t talk yourself out of the boring option.
- Tail the transaction log via change data capture — the approach tools like Debezium take, reading the database’s own write-ahead log and emitting a message per committed outbox row. Far lower latency and no polling load, at the cost of a meaningful piece of infrastructure to run and understand.
Start with polling. Reach for log tailing when the latency or the query load actually hurts — and write down why you moved, because it’s the kind of choice a future maintainer will want the reasoning for. (This is precisely what an architecture decision record is for.)
The honest summary
The outbox pattern doesn’t make distributed writes atomic — nothing does. What it does is refuse the dual write entirely: it collapses “save and publish” back into a single local transaction, and then makes delivery a separate, retryable, at-least-once problem that you can actually solve. You trade a silent data-loss bug you can’t see for an extra moving part you can monitor and a duplicate-delivery problem you can handle with idempotency.
That’s a good trade. It’s also the point where you have to stop believing the broker will deliver each message precisely once — because it won’t, and the next post is about why that’s fine.