The distributed systems tax post ended on a line I want to pick up: once a request crosses a network boundary, “what happened here?” stops being a stack trace and becomes a correlation ID, a tracing tool, and a log search across a dozen services. I listed it as one item on the bill. It deserves more than a line, because it’s the item most teams pay late — after go-live, in an incident, at the worst possible moment to discover they can’t.
Here’s the uncomfortable bit: you cannot attach a debugger to production. The system that’s misbehaving is the one you can’t step through, can’t pause, can’t inspect. Everything you will ever know about a live failure is whatever the system chose to emit before it went wrong. Observability is the decision about what that “whatever” is — and like most of the properties that matter, it’s one you make at design time or not at all.
Monitoring tells you that; observability tells you why
The words get used interchangeably, and the difference is the whole point.
Monitoring answers questions you knew to ask. Is the CPU hot, is the queue deep, is the error rate above the line you drew? You build those dashboards up front because you can already name the failure. That’s known-unknowns — you know what you’re looking for, you just don’t know when it’ll happen.
Observability is whether you can answer a question you didn’t anticipate, after the fact, without shipping new code to go and find out. Why is this one customer’s journey slow, but only on Tuesdays, but only when they hit the discount path? Nobody built a dashboard for that. You either have the raw material lying around to reconstruct it, or you’re pushing a logging change to production and waiting for it to happen again. That’s the line between the two: monitoring is the questions you prepared for, observability is the ones you didn’t.
The raw material is the familiar trio, and it helps to be honest about what each is actually for:
- Traces — the journey of a single request across every service it touched. This is the signal distribution makes essential; it’s the stack trace you lost when the call left the process.
- Metrics — cheap, aggregated, alertable numbers. The “is it bad right now” signal, and the one you hang alerts off.
- Logs — the detail, expensive at volume, invaluable when you need it. Structured (proper key-value fields) or don’t bother — a wall of free text you can’t query is a diary, not a tool.
The real unlock isn’t any one of them. It’s correlation: one request ID that stitches a trace to its logs to the metric spike, so you can pivot from “the error rate jumped” to “these forty requests, this service, this line” in one move. And correlation doesn’t happen by accident. It happens because every hop propagates the context — which is a design decision, not a feature you switch on.
Designed in, not bolted on
This is where observability stops being an ops concern and becomes an architectural one. It’s an -ility — one of the non-functional requirements that are the architecture — and like all of them, it’s brutal to retrofit and cheap to design in.
Cheap to design in, because the things that make a system observable are decisions you take early and then stop thinking about:
- Context propagation has to be pervasive. A trace is only as complete as its least-instrumented hop; one service that drops the context breaks the chain for everything downstream of it. This is an all-or-nothing property, which is exactly why it has to be a platform default and not a per-team afterthought.
- Structured logging is a contract. Decide the shape — correlation ID, service, level, event —
once, early, and hold everyone to it. Agree the semantic conventions: if half your services call
it
user_idand the other halfuserId, you’ve got the data and still can’t join on it. - Instrument through a standard, not an agent. This is where OpenTelemetry earns its keep — you instrument your code once against an open standard and decide later which backend consumes it. That keeps the expensive decision a two-way door: swapping vendors is a config change, not a re-instrumentation project. A proprietary agent wired through your code is the one-way door you’ll regret the first time the bill or the feature set disappoints.
None of that is work you can meaningfully do “later.” Later means threading context through code that was never built to carry it, renaming fields across services that are already in production, and unpicking an agent that’s grown into everything. Retrofitting observability is its own migration project, and it always lands mid-incident, which is the one time you have no slack to do it.
The bit that’s genuinely hard to get right
Being first-class doesn’t mean maximal. The failure mode at the other end is real: instrument everything, keep every trace, label every metric with every field you can think of, and you get a system that’s expensive to run, slow on the hot path, and drowning in signal nobody reads. There’s a cost to observability, and “turn it all on” is not a strategy — it’s a bill you haven’t opened yet. Deciding how much signal is worth paying for is a whole discipline of its own, and it’s where this thread goes next.
For now, the bar for “first-class” is simpler than maximal and harder than default: a new service ships with context propagation on, structured logs carrying a correlation ID, the handful of metrics that actually describe its health, and a deliberate answer to “how much of this do we keep.” Not because observability is a virtue you accumulate, but because the alternative is finding out, at 3am, across six services, that the only thing you can see is that something is wrong — and never why.