The last two posts were about mechanics: the outbox pattern to stop losing messages, and idempotency to survive the duplicates that at-least-once delivery guarantees. Both leave you with the same consequence, and it’s the one that’s hardest to explain to the people who sign off the work: for a while — usually milliseconds, occasionally longer — the different parts of your system will hold different versions of the truth. This is eventual consistency, and the engineering is the easy half. The hard half is the conversation.
Why the conversation goes wrong
An engineer says “the systems are eventually consistent” and hears “correct, with a small delay.” A stakeholder hears “sometimes the data is wrong.” Both are describing the same behaviour; only one of them is in the room when the trust is lost.
The mistake is defending the idea in engineering terms — CAP theorem, partition tolerance, replication lag. All true, all useless in that meeting. Nobody approving a payments change cares that you can’t have consistency and availability under a network partition. They care whether a customer can be charged twice, or shown a balance that’s a lie.
So don’t explain the mechanism. Explain that the business already runs on eventual consistency, and has done since long before there were computers to blame.
The business already works this way
Reach for an example from the room’s own domain, because the abstraction only lands when it’s wearing familiar clothes.
- A bank transfer. You send money at 4pm; it appears in the other account tomorrow. For a while, that money exists in neither place, or in both. Nobody thinks the bank is broken. There’s a well-understood period where the world is settling, and a well-understood guarantee that it will.
- An insurance policy. You buy cover online and you’re covered from that moment — but the documents arrive by post next week, the underwriting system reconciles overnight, and the premium-finance agreement is set up on its own cycle. The customer is protected immediately; the system agrees with itself over the following days. No one calls the gap a defect. It’s how the business has always operated.
- An invoice. Sent today, paid in thirty days, reconciled when the payment clears. Three systems, three different moments, one eventually-consistent truth about whether the bill is settled.
None of these is “wrong data.” Each is a known, bounded period of disagreement with a guaranteed resolution. That phrase is the whole point — and it’s language a business person already trusts, because they’ve been managing exactly that kind of gap their entire career. Your distributed system isn’t introducing a strange new risk. It’s doing, in software, what the organisation already does on paper.
The part you do have to promise
Here’s where honesty matters, because eventual consistency is easy to abuse as a shrug. “It’ll be consistent eventually” is only reassuring if you can answer two questions the business is right to ask:
- How long is “eventually,” at the worst? “A few hundred milliseconds normally, and under ten seconds even when a downstream system is struggling” is a commitment you can hold and monitor. “Dunno, depends” is not eventual consistency — it’s just inconsistency with better marketing. Put a bound on it, and alert when you breach it.
- What does the customer see during the gap? This is a design decision, not an accident. You can show a pending state (“payment processing”), you can optimistically show the end state and quietly reconcile, or you can block until it settles. Each is a legitimate choice with a different trade-off between responsiveness and the risk of showing something that later changes. The wrong move is to not decide, and let the gap surface itself as a confusing flicker.
Answer those two, and “eventually consistent” stops sounding like an excuse and starts sounding like what it is: a managed, bounded, monitored property of the system.
When the gap needs actively unwinding: sagas
Sometimes a business action spans several systems and can’t be one transaction — reserve stock, take payment, book dispatch, each owned by a different service. If payment fails after stock is reserved, you can’t roll back across all of them the way a single database would. There was never a shared transaction to roll back.
The pattern here is the saga: a sequence of local steps, each of which knows how to undo itself with a compensating action. Not a rollback — a deliberate reversal. And again the business already has the vocabulary, because the physical world has always had this problem:
- You don’t “roll back” a dispatched parcel — you issue a return.
- You don’t “roll back” a payment — you issue a refund.
- You don’t “roll back” a mistaken insurance document — you send a cancellation letter.
A compensating action is the software version of a refund or a cancellation: an explicit, recorded, forward step that makes amends for one that can’t be taken back. Framed that way, a finance stakeholder doesn’t need the word “saga” at all. They need to know that if step three fails, steps one and two are actively reversed by real operations that leave an audit trail — which is precisely how the business would expect a half-finished transaction to be handled anyway.
The honest summary
Eventual consistency isn’t a weaker kind of correctness you’re apologising for. It’s the recognition that distributed systems settle the same way organisations always have — with a known, bounded, monitored period of disagreement and a guaranteed resolution, plus explicit compensating actions when a multi-step process has to be unwound.
The engineering — outbox, idempotency, sagas — is well-trodden. The skill that actually protects the project is translating it out of CAP-theorem language and into the language of money, letters, and obligations, where the people funding the work already understand it intuitively. Do that, and “eventually consistent” stops being the phrase that loses their trust and becomes the one that earns it.