Observability without noise: metrics, logs, and traces that are actually used

Storing everything is easy. The problem appears when you have to open a dashboard at three in the morning and find, among thousands of series, the one that explains why a service stopped responding. In most of the platform teams we review, the volume of metrics, logs, and traces grows faster than the actual ability to interpret them. And the cost of storage, sooner or later, starts to weigh more than the usefulness.

A criterion that works: classify each signal by its operational value, not by how cheap it is to emit. The metrics used in a specific incident—latency per endpoint, error rate per dependency, saturation of a connection pool—deserve long retention and controlled cardinality. Debug logs that no one consults after deployment can live for hours, not weeks. Distributed traces are useful when there is a jump between services that cannot be explained with metrics; outside that case, sampling is reasonable.

On real dashboards this translates into a few well-chosen series. A dashboard with twenty charts is usually less useful than one with five, each answering a specific question: is latency rising, is some resource running out, is a dependency failing more than yesterday? When the panel tries to cover everything, the operator wastes time deciding what to look at first. That prior selection is the team's job, not the on-call person's.

Regarding traces, the discussion repeats: store 100% or sample? It depends on traffic and on how much you need to reconstruct a specific case. Rate-based sampling, combined with full capture when there is an error, covers most diagnostics without inflating the backend. What does not work is sampling without criteria and discovering in the middle of an incident that the trace you needed was not stored.

Retention decisions are also cost decisions. Defining how long each type of signal lives —and who can change it— prevents storage from growing out of inertia. There is no single formula; there are habits: periodically reviewing what is being stored, what is actually queried, and what can be discarded without losing response capability.