The last post left you with a loose thread. The outbox pattern gives you at-least-once delivery: the relay can publish a message and then die before it records the row as sent, so on restart it publishes again. I said the fix was to make your consumers idempotent, and that “exactly once” was a lie you’d be tempted to believe. Time to pull that thread.
What people mean when they say “exactly once”
They mean: this message will be delivered to my consumer, and the consumer’s effect will happen, one time — no more, no less. It’s a completely reasonable thing to want. It is also, taken literally, impossible over an unreliable network, and it’s worth understanding why rather than just being told to stop asking.
Picture the relay publishing a message to the broker. It sends the bytes and waits for an acknowledgement. One of two things can go wrong, and from the sender’s side they are indistinguishable:
- The broker never received the message, so no ack is coming.
- The broker received it fine, but the ack was lost on the way back.
This is the Two Generals’ Problem, the oldest impossibility result in distributed computing. It’s why none of this is a bug waiting to be fixed. Two generals must agree to attack together, but can only coordinate by sending messengers across a valley the enemy holds — and any messenger, including the one carrying the acknowledgement, might not make it. No number of confirmations-of-confirmations ever makes both certain they agree, because the last message sent is always the one nobody confirmed. Your relay and your broker are those two generals; the network is the valley. First framed in the 1970s, it has never been solved, because it can’t be.
So the sender is stuck with a genuine dilemma. If it doesn’t resend, it risks losing a message that never arrived. If it does resend, it risks delivering a message that already arrived twice. There is no third option, because the sender cannot tell the two cases apart. This is the distributed systems tax in its purest form: partial failure means you are always choosing which way to be wrong.
Every messaging system resolves this the same way. It picks “resend” — because losing data silently is the worse failure — and calls the result at-least-once. Duplicates aren’t a bug in the broker; they’re the price of not losing messages.
So what is “exactly-once delivery” being sold as?
You’ll see brokers advertise “exactly-once.” Read the small print and it’s almost always one of two honest things wearing a bolder label:
- Exactly-once processing within a closed system. Kafka’s exactly-once semantics, for instance, are real — but they hold inside Kafka: consume from a topic, produce to a topic, commit the offset, all in one transaction owned by one system. The moment your side effect leaves that world — you charge a card, send an email, call someone else’s API — the guarantee stops at the boundary. It can’t reach into a payment provider it doesn’t control.
- At-least-once plus deduplication, presented as one feature. Which is exactly the thing you’re about to build yourself — and building it yourself means you actually understand where it holds and where it doesn’t.
Neither is wrong. What’s wrong is reading the label and concluding you don’t have to think about duplicates.
The honest version: at-least-once + idempotency
You can’t stop duplicates arriving. What you can do is make a duplicate a non-event — process the same message twice and land in the same state as processing it once. That property is idempotency, and it’s the real deliverable. “Exactly once” isn’t something the network gives you; it’s something your consumer earns by being idempotent about an at-least-once stream.
There’s a distinction worth keeping sharp here. Some operations are naturally idempotent —
“set the status to shipped” lands in the same place however many times you run it. The ones that
hurt are the accumulating effects: “charge £40,” “add a loyalty point,” “send the dispatch
email.” Run those twice and you’ve taken £80 and annoyed a customer. Those are the operations that
need protecting, and there are a few honest ways to do it.
A deduplication table, keyed by message ID. Give every message a stable, unique ID at the point it’s created — in the outbox pattern, that’s the outbox row’s primary key, which never changes across redeliveries. The consumer records processed IDs in its own database and, in the same transaction as the side effect, checks whether it’s seen this one before:
using var txn = _db.BeginTransaction();
if (await _db.HasProcessed(message.Id)) // seen it → the duplicate is a no-op
return;
await ApplyEffect(message); // the real work
await _db.MarkProcessed(message.Id); // same transaction as the effect
txn.Commit();The commit is the trick — the same single-database atomic commit the outbox relied on. The effect and the record-of-having-done-it either both land or both roll back. There’s no window where you’ve done the work but forgotten you did it.
A natural idempotency key you already own. Often you don’t need a generic dedup table because the domain hands you a unique key: an order ID, a payment reference, an agreement number. A unique constraint on that column turns a duplicate insert into a caught error instead of a second row. The database enforces once-ness for you — lean on it.
An idempotency key passed downstream. When the side effect is a call to someone else’s
system, you can’t dedup it in your database — the second call has already left the building. The
move is to push the key across the boundary: good payment APIs accept an Idempotency-Key header
precisely so that a retried charge with the same key is recognised and collapsed on their side.
You’re delegating the dedup to the only party who can actually enforce it — the one holding the
side effect.
Where this bites, and the caveats
A dedup table is not free, and pretending otherwise would repeat the exact sin this post is about.
- It grows. Same as the outbox — processed-ID records need a retention window and a cleanup job. You can usually prune aggressively: once a message is old enough that the broker will never redeliver it, its dedup row has done its job.
- Idempotency has a scope in time. “Have I processed this ID?” is only answerable while you still remember the ID. Prune too eagerly and a very late redelivery sails through as new. Match the retention to the broker’s maximum redelivery window, not to a number that felt tidy.
- The key has to be stable. If a message’s ID is regenerated on each publish attempt, your
dedup table is worthless — every duplicate looks new. This is the same
?? Guid.NewGuid()instinct that fabricates a key rather than admitting it’s missing: a fresh ID per attempt quietly defeats the whole mechanism. The ID must be minted once, with the message, and carried unchanged through every redelivery.
The honest summary
“Exactly-once delivery” is a promise about the network that the network cannot keep. What you can build — and what the brokers advertising the phrase are quietly doing on your behalf — is at-least-once delivery plus idempotent processing. The duplicates still arrive; you’ve just made them harmless.
That’s not a downgrade from the dream. It’s the dream, correctly specified: stop asking the delivery layer for a guarantee it can’t give, and put the guarantee where you can actually enforce it — in your own transaction, keyed by an ID you control. Like most things, the guarantee doesn’t come from a feature you switch on; it comes from a decision about where the once-ness lives.
Which leaves one more uncomfortable truth to face: even with all this, your systems will be temporarily out of step with each other, and that’s not a failure to engineer away — it’s a condition to design for. That’s the next post.