Monitoring answers questions you thought of in advance. Observability lets you answer questions you did not. The distinction sounds academic until an incident involves a failure mode nobody anticipated, which is most of them.
The practical difference
Monitoring is dashboards and alerts built around known failure modes. CPU above 90%, error rate above 1%, queue depth above a threshold. It is essential and it is bounded by imagination.
Observability is the property of a system that lets you investigate novel problems from its outputs, without shipping new code to add instrumentation. “Why are requests from this one customer’s tenant slow, but only on uploads over 10MB, and only since Tuesday” is an observability question. No dashboard was built for it.
You need both. Monitoring tells you something is wrong; observability tells you what.
The three signals
Metrics are numeric measurements over time — request rate, latency percentiles, error counts, queue depth. Cheap to store, fast to query, excellent for alerting and trends. They aggregate away detail, so they tell you something is slow but not which requests.
Logs are discrete event records. Rich in detail, expensive at volume, and only useful if structured. A log line that is a sentence is for humans reading one at a time; a log line that is a JSON object with consistent fields can be queried.
Traces follow a single request across services, showing where time was spent. In a distributed system this is the signal that turns a four-hour investigation into a ten-minute one, and it is the one most teams add last.
What makes a system observable
Four properties, roughly in order of impact.
Structured logs with consistent fields. Request ID, user ID, tenant ID, operation name on every line. This single change makes logs searchable rather than readable.
A correlation ID propagated everywhere. One identifier, generated at the edge, passed through every service call and attached to every log line. Without it, reconstructing a request’s path across services is guesswork.
High-cardinality attributes. The ability to slice by customer, by endpoint, by version, by region. Most hard problems affect a subset, and a system that can only show you aggregates cannot find a subset.
Instrumentation at boundaries. Every outbound call — database, cache, third-party API — timed and recorded. The majority of latency problems live in a dependency.
Cardinality, and why it costs so much
Cardinality is the number of distinct values an attribute can take. Status code has low cardinality — a handful of values. User ID has high cardinality — one value per user.
Traditional metrics systems store a separate time series per combination of label values, so adding a user ID label to a metric with ten thousand users creates ten thousand series. Add endpoint and region and the multiplication becomes brutal. This is the single most common cause of an unexpectedly enormous observability bill.
The practical rules: keep high-cardinality attributes out of metrics and put them in traces and structured logs, where storage is per-event rather than per-combination. Use exemplars to link an aggregate metric to a specific trace. And review your label sets periodically, because they accumulate.
Sampling
At any real volume, storing every trace is unaffordable. Sampling is how you keep the useful ones.
Head sampling decides at the start of a request, usually keeping a fixed percentage. Simple, cheap, and it discards most errors because errors are rare.
Tail sampling decides after the request completes, so you can keep everything that errored or was slow and a small sample of the rest. More expensive to operate and far more useful.
A common and sensible policy: keep 100% of errors, 100% of requests over a latency threshold, and 1–5% of everything else.
Alerting on symptoms, not causes
The most common alerting failure is pages that fire on causes — high CPU, memory pressure, a full disk — most of which do not affect users.
Alert on what users experience: error rate, latency at a high percentile, and whether critical workflows complete. If CPU is at 95% and nobody notices, that is a capacity planning item for Monday, not a page at 3am.
Every page should be actionable, urgent and about something real. An alert that routinely resolves itself trains people to ignore alerts, which is how a genuine incident gets missed.
Controlling cost
Observability bills grow faster than traffic if nobody manages them. Five levers.
- Sample traces aggressively while keeping all errors.
- Set log levels deliberately. Debug logging in production at full volume is the most common source of surprise ingest charges.
- Tier retention. Thirty days hot and queryable, a year in cheap archive storage. Most investigation happens within days.
- Prune unused metrics. Metrics are easy to add and nobody removes them. Audit annually.
- Drop noisy events at the collector rather than paying to ingest and then ignore them.
The same discipline that applies to cloud cost applies here, and observability is often the second-largest line after compute.
Where to start with a small team
In order, each useful on its own.
- Structured logging with a correlation ID. Cheap, and it transforms debugging.
- Four service-level metrics per service: request rate, error rate, latency percentiles, saturation.
- Alerts on error rate and latency only. Add more when you have evidence you need them.
- Distributed tracing once you have more than two or three services.
- A dashboard per service that answers “is this healthy”, not forty panels nobody reads.
Instrumenting with a vendor-neutral standard rather than a proprietary agent is worth the small extra effort, because changing vendors otherwise means re-instrumenting everything.
Dashboards people actually use
Most organisations have dozens of dashboards and consult three. The useful ones share a structure.
One service overview per service, answering a single question: is this healthy right now? Four or five panels — request rate, error rate, latency percentiles, saturation — and nothing else. If it takes more than five seconds to read, it is the wrong dashboard for an incident.
One incident dashboard showing the system end to end: traffic at the edge, errors by service, dependency health. This is the first thing opened when something is wrong and the last thing anyone wants to be building during an outage.
Deep dives on demand. Everything else belongs in ad-hoc queries rather than a saved dashboard, because the questions during a novel incident are novel by definition.
Delete dashboards nobody has opened in three months. They cost attention and they are usually wrong by the time someone does open them.
What to capture during an incident
Observability is only as good as the use made of it afterwards. Four things worth recording while an incident is live, because reconstructing them later is unreliable.
- The timeline of what was observed and when, including the first signal and how it was noticed.
- Which queries or dashboards helped and which were dead ends — the dead ends tell you what instrumentation is missing.
- Anything that had to be guessed because the data was not there.
- The deploy, config change or traffic shift that preceded it.
The second and third items are the ones that improve the system. A review that produces “we should add tracing to the payments path” from a concrete dead end is worth more than a general resolution to improve observability.
Instrumenting the edges
The boundaries of a system produce the most useful signals and are the most commonly under-instrumented. Every outbound call should record duration, status and the identity of the dependency, so a slow dependency is visible without guesswork.
This applies to inbound boundaries too. An API that records which client made each call, at what rate and with what error profile gives you both an operational view and the data needed to set sensible limits — see API rate limiting for how those two uses connect.
Frequently asked questions
Do I need all three signals?
Metrics and structured logs are the minimum. Traces become important once requests cross service boundaries, and before that they add less than the effort costs.
What is an SLO?
A service level objective is a target for a user-facing measure, such as 99.9% of requests succeeding over 30 days. Alerting on the rate at which you are consuming the error budget is more useful than alerting on instantaneous thresholds.
Should logs or traces come first?
Logs, because they are cheaper to adopt and useful immediately. Traces deliver more per incident but require instrumentation across every service to be worth much.
How long should data be retained?
Thirty days of queryable data covers most investigation. Longer retention in cheaper storage is worth having for trend analysis and for any compliance requirement.
