Here’s a claim I’d defend as a platform-wide default rather than a technique you reach for on frightening changes: anything replacing an existing implementation should be capable of running alongside the thing it replaces, on real traffic, producing outputs you can compare, with every real-world effect switched off.
That’s shadow mode. New path runs, does the work, produces an answer. The old path stays authoritative and its answer is the one that reaches the customer. You compare the two, measure how often and how much they diverge, and only when the divergence is understood do you flip which one counts.
The reason I’d make it a default rather than a special measure is uncomfortable but simple. The moment you reserve it for the scary changes, you’ve introduced a judgement about which changes are scary — and that judgement is precisely the thing most likely to be wrong. Nobody schedules the incident.
Contract tests prove the shape; this proves the behaviour
Contract testing ended on an honest admission: it buys compatibility, not correctness. A response can satisfy a contract to the letter — every field present, every type right — and still be completely wrong. Right shape, nonsense values.
Shadow mode is the thing that closes that gap, and it’s why I think of the two as a pair rather than alternatives. A contract test asks does this conform to the agreed shape? Shadow mode asks does this produce the same answer the trusted implementation produces, on the actual traffic customers are actually sending? You want both, and neither substitutes for the other: a shadow run can’t tell you that you’ve broken a consumer’s expectation of the schema, and a contract test can’t tell you that your new pricing logic is out by a penny on every third case.
It’s also a different instrument from a spike. Design up front argued for building something small to buy information you can’t reason your way to. A spike buys information about whether an approach is viable. Shadow mode buys information about whether an implementation is right, and it does it against ground truth you couldn’t manufacture in a test environment — real inputs, in their real distribution, including the ugly historical ones nobody would think to write a fixture for.
Side-effect suppression is the whole difficulty
The comparison is the easy half. The hard half — where this either works or embarrasses you — is that the shadow path must do its work while touching nothing in the real world.
Concretely, everything in this list has to be identified and gated before you dual-run:
- No customer communication. No emails, no letters, no notifications. This is the one that reaches a human fastest and the one people notice.
- No money movement. No payments taken, no refunds issued, no collections submitted.
- No third-party calls with consequences. Anything that books, binds, reserves, notifies or bills an external party. Read-only lookups are usually fine; anything that mutates state at the other end is not, and the boundary between those two is not always documented honestly.
- No writes to authoritative state. The shadow path can persist its own output for comparison, but it must not update the records the live path owns.
The reason to enumerate this rather than wave at it: side effects are not evenly distributed through a codebase, and the one you miss is never the one you were thinking about. It’s the audit-log write that also triggers a downstream consumer, or the “harmless” call that increments a usage counter someone bills against.
Ports and adapters make this dramatically easier, which is one more argument for the seam that anti-corruption layers work both ways describes. If every outbound effect already goes through an explicit port, suppression is a configuration decision at the adapter. If effects are scattered inline through your handlers, you’re going hunting, and you’ll find out how well you hunted from a customer.
The route out is a switch, not a project
Shadow mode is most of the value, but it earns the rest of its keep by being the first position on a ramp you can walk without deploying: shadow, then a canary slice of real traffic where the new path does count, then wider, then full. And backwards just as fast.
That wants feature flags rather than branches — the whole point is that changing which implementation is authoritative is a runtime decision, made in seconds, by someone watching a dashboard. If flipping to the new path requires a release, you don’t have a ramp; you have a series of small big-bangs.
Combine it with a strangler migration and the cutover stops being an event. Each capability moves shadow → canary → authoritative on its own timeline, and the tracking mechanism that decides which system serves a request is the same mechanism that rolls it back. That’s the difference between “we migrated on the 14th” and “we stopped needing the old one at some point in March”, which is a much better kind of sentence.
Divergence is a question, not a defect
Now the honest turn, and it’s the part I’d most want to say out loud, because it’s the thing that makes shadow mode harder than it sounds.
Shadow mode does not prove the new implementation is correct. It proves it agrees with the old one. Those are very different claims, and the gap between them is where the interesting failures live.
If the incumbent has a bug — and an incumbent that has been in production for a decade certainly does — then a faithful new implementation will diverge, and every instinct in the process will read that as a defect in the new code. I have watched people “fix” a correct implementation to match an incorrect one, because the comparison report said red and red means fix. Sometimes the right response to divergence is to change the old system, or to accept the difference and tell someone, or to decide the old behaviour was load-bearing for customers and reproduce the bug on purpose.
So every difference needs adjudicating by somebody who knows which answer should be right. That’s the cost nobody budgets: not the compute, not the comparison harness, but the human judgement on a list of differences, repeatedly, for weeks.
The honest limits
- Defining “same” is a real design problem. Timestamps, generated identifiers, collection ordering, floating-point rounding and anything derived from “now” will differ legitimately every single time. Without a deliberate comparison policy — which fields matter, what tolerance is acceptable, what’s excluded — you get a report that is always red, and a report that is always red gets ignored within a fortnight.
- You pay for two of everything while it runs. Double the compute and double the telemetry, on the observability bill you were already looking at nervously. Time-bound it: a shadow run with no planned end becomes another permanent parallel system.
- It needs an incumbent. Greenfield has no ground truth to compare against, so this isn’t available for the thing you’re building for the first time — which, awkwardly, is also when you’d most like reassurance.
- Divergence observability is not optional. If you can’t see the comparison — rates, magnitudes, which cases, trending over time — you haven’t built shadow mode, you’ve built a second copy of the work that nobody reads. Same argument as observability as a first-class concern, and the same failure if you skip it.
- Suppression can hide a real problem. If the new path’s side effects are switched off, you are explicitly not testing them. The first time it runs authoritative is the first time anything downstream of it happens for real — so a canary slice isn’t optional polish, it’s the only stage where the effects get exercised at all.
The one-line version
Run the new thing next to the old thing, on real traffic, with everything that touches the world switched off, and compare. Contract tests tell you the shape is right; only a parallel run against real inputs tells you the answers are. And when they differ, resist the assumption that the new code is the one that’s wrong — the whole exercise is worth most at exactly the moment it tells you something you didn’t want to hear.