Observability Wiki

Metrics

RED Method vs. USE Method

Two complementary frameworks for deciding what to measure: RED for services, USE for resources.

Last updated

RED and USE are two frameworks for deciding which metrics actually matter, out of the near-infinite set you could collect. Both were popularized in the mid-2010s as reactions to teams drowning in dashboards full of low-value data, and both work by narrowing focus to a small, fixed set of signals per thing being monitored.

They apply to different layers of a system, which is why they’re usually used together rather than as alternatives.

The RED method

RED was introduced by Tom Wilkie (then at Weaveworks) as a model for instrumenting request-driven services, adapting Google’s “four golden signals” into a tighter, service-focused set. For every service, you track:

  • Rate — the number of requests the service is handling per second.
  • Errors — the number of those requests that are failing, whether by exception, HTTP 5xx, or an application-level failure code.
  • Duration — the distribution of time those requests take, typically expressed as percentiles (p50, p95, p99) rather than an average.

RED is measured from the caller’s point of view, which makes it consistent across every service in an architecture regardless of what the service does internally — a payments service and a search service both get rate, errors, and duration, and the resulting dashboards look the same shape everywhere.

The USE method

USE was defined by Brendan Gregg for diagnosing performance problems in resources — CPU, memory, disk, network interfaces, and other components that can become a bottleneck. For every resource, you track:

  • Utilization — the percentage of time the resource was busy servicing work, or the percentage of its capacity in use.
  • Saturation — the amount of work queued that the resource couldn’t service immediately, such as run-queue length or a request backlog.
  • Errors — the count of internal error events for that resource, like disk I/O errors or dropped network packets.

USE is designed for a quick, systematic sweep across every resource in a system to find the one that’s actually constrained, rather than guessing.

Why it matters

  • RED tells you a service is unhealthy; USE tells you why. A spike in RED’s error rate or duration for a checkout service points you at the symptom. Following up with USE metrics on that service’s host — CPU utilization, memory saturation — often reveals the underlying resource constraint causing it.
  • Both frameworks force restraint. Without a model, teams tend to either instrument everything (creating cardinality and cost problems, see Cardinality and Cardinality Explosion) or instrument nothing consistently. RED and USE each cap the signal set at three per target, which keeps dashboards legible.
  • They compose across a full stack. Applying RED at every service boundary and USE at every host, container, and resource gives uniform coverage from the top of a request’s path down to the hardware, without requiring a bespoke dashboard design for each component.
  • They map cleanly onto PromQL and similar query languages. Rate is a natural fit for rate() over a counter, errors are a ratio of two counters, and duration percentiles come from histogram buckets — the same query shapes recur across every service dashboard.

How to apply them together

  • Start with RED at the edges. Instrument every service entry point (HTTP handler, gRPC method, queue consumer) with request counters, error counters, and a duration histogram.
  • Apply USE to everything RED depends on. CPU, memory, disk I/O, network, connection pools, and any queue or worker pool get utilization, saturation, and error tracking.
  • Use RED to detect, USE to diagnose. Alerting typically lives on RED signals (error rate too high, p99 duration too slow) because they reflect user-visible impact; USE metrics are usually where you look next during investigation.
  • Extend USE conceptually to non-hardware resources. Connection pools, thread pools, and rate limiters behave like resources too — utilization (how full), saturation (queue depth), and errors (rejections) apply just as well to them as to CPU or disk.

Together, RED and USE give a full-stack observability strategy without requiring a metric for every conceivable thing: a small, consistent shape for services, and a small, consistent shape for the resources underneath them.

Related tools