A customer-facing journey in one of our pre-production environments stopped working on a Saturday. It was still broken the following Wednesday.
The cause, once someone looked, was mundane in the best way: a service populates a reference cache once at startup, and that population had failed. No retry, no refresh, no second attempt — just an empty cache, quietly turning every subsequent request into a wall of validation failures. Nothing was down. Everything was responding. It was all responding wrongly.
Here’s the part that has stayed with me. The health check for that service includes cache state, and it had been red for days.
So this isn’t a story about a gap in the monitoring. Someone had done the good work — thought about what “healthy” means for that service, decided that an empty reference cache doesn’t qualify, and wired it into the check. That’s exactly right, and it’s more care than most services get. The signal existed, it was accurate, it was specific, and it pointed directly at the cause.
Nobody read it.
Two kinds of alert, two ways to be ignored
A colleague made the distinction that unlocked this for me, and it’s the same cause-and-effect axis I used a couple of weeks ago to argue about who owns the alert — arrived at from the opposite direction. There are at least two kinds of alert in play:
- The technical system has encountered a problem. The cause.
- The users of the system are experiencing a degraded service. The effect.
In production, both have someone attached. In a non-production environment, both fail — and they fail for completely different reasons.
- Cause alerts fail by volume. The tech team fatigues, and my colleague’s diagnosis of why was refreshingly blunt: partly because we don’t engineer well enough to avoid problems in the first place, and partly because we’ve decided a test environment is allowed to be broken. Once breakage is expected, it’s tolerated; once it’s tolerated, the alert carries no information. You haven’t tuned the noise out, you’ve trained yourself out.
- Effect alerts fail because there’s nobody to care. Almost nobody watches “users are getting a degraded service” in a test environment, for the very good reason that there are no users. And this one isn’t a culture problem or laziness — it’s structural. The effect only has an owner when there’s a customer, so in non-production there is no process owner watching the outcome, because the role doesn’t exist there.
That second one is the bit I keep turning over. The effect alert is the one that matters most in production, and it is the one you get zero practice at beforehand. Every other muscle gets exercised in pre-production — deployments, rollbacks, load, failover. The habit of noticing that customers are having a bad time is the one thing you cannot rehearse there, and so the first time anyone does it for real is the first time it’s real.
The only difference that mattered
I asked what, structurally, separates an alert that gets acted on from one that doesn’t. In this case the answer was embarrassingly simple: production alerts get logged into the ticket queue, where someone has to action them. Non-production alerts don’t.
Not severity. Not accuracy. Not how good the instrumentation is — we’ve established the instrumentation was good. The difference was whether the signal landed somewhere with a workflow and a name attached.
Which is the same thing I’ve been arguing for three posts now, in its narrowest and most practical form: an alert without an owner and a workflow isn’t an alert, it’s a notification. Ownership is not a technical decision at the routing layer, and it isn’t one here either. A red health check on a dashboard is information sitting in a room. Whether anybody is in the room is an organisational fact, not a technical one.
So route everything into the queue? No
The obvious fix is to send non-production alerts to the ticket queue too, so they land somewhere with teeth. Someone in the thread immediately asked the right question: and would we get queue fatigue?
Yes. Obviously yes. Fatigue is conserved. Routing alerts into a workflow doesn’t destroy the noise, it relocates it — from a channel people have learned to scroll past to a queue people learn to bulk-close. You’ve moved the ignoring, added ceremony to it, and given yourself a metric that looks like process.
This is the same trap as the observability bill: the instinct that more signal is more safety, when past a certain point additional signal is negative value because it degrades the signal you already had. There is no routing configuration that fixes an alert-to-noise ratio. The only real cure is fewer, truer alerts — which means the uncomfortable work of deleting alerts that have never once caused an action, and fixing the underlying flakiness that generates the rest.
The reframe: it’s an early warning system, not a playground
The thing that actually shifts this, I think, is refusing the premise that a test environment is a place where breakage doesn’t count.
It’s the same setup as production. It often takes more traffic than production, because synthetic checks hammer it continuously. Which makes it an early warning system as much as a playground — and it means the alert volume is not a nuisance to be tuned away. The volume is the finding. If a pre-production environment generates a lot of alerts, the honest reading isn’t “these alerts are noisy”, it’s “this system breaks a lot and we’ve built somewhere comfortable to not notice”.
Treated that way, the cache failure was a gift. It found a latent single point of failure — a cache populated once at startup with no retry is a time bomb with no counter on it — and it found it in the one environment where nobody was harmed. That’s the system working. The failure wasn’t the outage. Detection worked perfectly; attention failed.
And “attention” is not a personal virtue you can exhort people into. It’s a function of whether someone is accountable for looking, which is the decision underneath all of this: a health check nobody owns is a health check nobody reads, however good it is. If you want the red one noticed, someone’s name goes against it — and, just as importantly, the green ones have to have earned their green, or the red is invisible in the crowd.
The line I can’t improve on
I’d been framing all of this as a problem of not holding pre-production to production’s standards. Then the same colleague reframed it, and his version is better than mine:
I think we’re not treating our test environment like production. I might say that we’re actually treating production like a test environment.
That’s the real risk, and it’s why this isn’t a hygiene issue about a lower environment. The habit of scrolling past a red check doesn’t stay where you learned it. You practise it every day in the place where it’s free, and then you carry it, fully formed, into the place where it costs you customers.
The cache was empty for four days. The check knew on day one.