Traditional monitoring answers questions you thought of in advance. Observability is about being able to ask new questions of a running system when something unexpected happens, which is when it matters.
The three signals
- Metrics: cheap, aggregated, good for alerting and trends
- Logs: detailed, expensive at volume, good for specific incidents
- Traces: request paths across services, good for finding where time goes
Structured logs are worth the effort
Log as structured data rather than free text, with a request identifier on every line. Correlating a user report to the exact request becomes seconds of work rather than an afternoon of grep.
Alert on symptoms
Alert on what users experience: error rate, latency, failed transactions. Alerting on CPU produces pages nobody can act on. Every alert should have an obvious first action or it should not exist.
Track deployments as events
Overlay releases on your dashboards. A large proportion of incidents begin with a deployment, and seeing the correlation immediately saves the first half of most investigations.
Practise
Run a drill: pick a plausible failure and find the cause using only your existing tooling. The gaps you hit are your roadmap.
The measure of observability is whether you can answer a question you did not anticipate, without shipping new code to find out.
Where to start
Add a request identifier to every log line and propagate it across services. It is the single highest-return change most systems can make.
