I argued a few weeks ago that observability is a first-class concern — something you design in, not bolt on. All true. But there’s a second half to that argument that the “instrument everything” crowd tends to skip, and it arrives as a surprise roughly a quarter after go-live: observability has a bill, and nobody reads it until it’s enormous.
The dashboard was free. The demo where you traced a request end-to-end and watched the spans light up was free. The invoice — for the data platform ingesting a firehose of telemetry, for the metrics backend buckling under a label you didn’t think about, for the p99 that crept up because every call now creates a span — turns up later, and by then “just turn it all on” has quietly become the architecture. This post is about opening the bill early, on purpose, while the numbers are still decisions and not emergencies.
Three ways the bill runs away
The costs don’t announce themselves. They compound quietly, each one a decision you didn’t realise you were making:
- Cardinality — the silent killer. This is the one that catches people. A metric isn’t one number; it’s one number per unique combination of its labels. Tag a metric with something low-cardinality — status code, region — and you have a handful of time series. Tag it with user ID, or request ID, or the full URL with its query string, and you have millions, each one stored and indexed separately. Cardinality, not raw request volume, is what actually detonates a metrics backend, and it does it as a step change: fine, fine, fine, then the bill has a comma you’ve never seen before. The decision hiding in here is what dimensions are genuinely worth slicing by — because every one you add multiplies, it doesn’t add.
- Volume and retention. You cannot keep every log line forever, and you don’t need to. But “how long is this actually useful” is a question most teams answer by default — infinite — rather than on purpose. A trace you’ll look at within the hour if at all, an access log you might need for an audit in a year, and a debug log that’s noise by lunchtime are three different retention policies, and paying to keep all three at the highest tier is how the storage line becomes the biggest one.
- Performance. Instrumentation isn’t free on the hot path. Creating spans, propagating context, serialising attributes, shipping it all to an exporter — individually tiny, collectively a budget nobody sets. It’s rarely the thing that takes you down, but it’s often the quiet tax that turns a fast path into a merely-acceptable one, and you won’t find it unless you go looking.
Sampling is a decision, not a failure
The instinct, when the bill lands, is to feel that sampling is giving up — that a real engineer keeps everything and anything less is cutting corners. It’s the same instinct that says more security is always better, and it’s wrong the same way.
Keeping 100% of your traces is neither affordable nor useful. The overwhelming majority of them record a request that did exactly what it was supposed to, and you will never look at a single one. The skill isn’t retention; it’s deciding what you’d regret not having and keeping that.
That decision has a shape worth knowing:
- Head-based sampling decides at the start of a request — keep one in a hundred, say — before you know how it turns out. Cheap and simple, but blind: it throws away the rare error at the same rate as everything else, so the one trace you actually wanted is probably the one you dropped.
- Tail-based sampling waits until the request finishes, then keeps the interesting ones — the errors, the slow ones, a thin sample of the boring rest. It costs more, because you have to buffer traces long enough to judge them, but it keeps the signal and bins the noise. For most systems that’s the trade worth making.
Either way you’re making an explicit choice about what’s worth seeing. “Keep everything” isn’t the absence of that choice — it’s the choice to pay maximum and decide nothing.
Optimise, don’t maximise
This is the same argument I made about how much security is too much, and it rhymes for a reason. Maximal observability has its own pathologies, and they look a lot like over-tightened security:
- Alert fatigue. Wire an alert to every metric and you don’t get more warning, you get more noise — and a team that’s learned to ignore the dashboard misses the one alert that mattered.
- Dashboard sprawl. Forty dashboards nobody trusts is worse than three that everyone reads, because the answer is in there somewhere and finding it costs you the incident.
- A bill that eats itself. The most expensive outcome isn’t the invoice — it’s that the invoice gets so alarming the next person rips observability out wholesale, and now you have none.
So the honest target, as with everything else on this blog, comes back to what you’re optimising for: not the most signal you can collect, but the appropriate signal for what the system is worth and how it fails. And if you want to put actual numbers on “appropriate,” you already have the tool — the risk equation works just as well on a telemetry spend as on a security control. What does it cost to collect this, what does it save you when something breaks, and is that trade positive? Signal that fails that test is noise you’re paying to store.
A budget you can defend
The move is to treat observability spend like any other architectural cost — deliberately, with an owner. Set a rough signal budget per service. Pick your sampling strategy on purpose and write down why. Look at cardinality before it bites, not in the post-incident review of the incident it caused. And put someone’s name against the bill, because observability nobody’s accountable for is the thing that grows unchecked until it’s cut in a panic.
The cheapest observability is the signal you decided you needed on purpose. The most expensive is the everything you collected because deciding felt like work — and then paid for, twice: once on the invoice, and once again the day someone turns it all off to make the invoice stop.