Observability beyond monitoring: Building a culture of operational excellence
Monitoring vs observability
Monitoring tells you that something is broken. Observability tells you why. The distinction matters because modern distributed systems have too many failure modes to预define every possible alert.
An observable system exposes enough internal state that you can diagnose novel failure modes without deploying new instrumentation. This is the fundamental difference between checking dashboards and investigating incidents.
The three pillars
Logs: Discrete events with context. Structured logging (JSON format with consistent fields) makes logs machine-queryable. We use a standardised format across all services: timestamp, level, service name, trace ID, and a structured message.
Metrics: Numeric measurements over time. The RED method (Rate, Errors, Duration) covers request-driven services. The USE method (Utilisation, Saturation, Errors) covers infrastructure. Use both.
Traces: The path of a request through your system. Distributed tracing connects logs and metrics to specific requests, making it possible to follow a single user action across ten services.
Culture over tools
The most common mistake is buying observability tools without changing practices. The tools are necessary but not sufficient. You also need:
- Runbooks: Documented procedures for every alert. If an alert does not have a runbook, it should not exist.
- Blameless post-mortems: Every incident produces a written analysis focused on systems and processes, not individuals.
- Observability reviews: Part of every PR review should be checking that new code includes appropriate logging, metrics, and tracing.
Getting started
You do not need to implement everything at once. Start with structured logging and distributed tracing. Add metrics dashboards for your most critical services. Build runbooks for your most frequent alerts. Expand from there.
The goal is not to monitor everything — it is to be able to ask arbitrary questions about your system's behaviour and get answers.