Traces

A trace represents the full journey of a single request as it passes through a distributed system — from the load balancer, through half a dozen microservices, to the database and back. Each unit of work within that journey is a span, and spans are linked together into a tree that shows exactly which service called which, how long each step took, and where time was actually spent.

Where a metric might tell you “p99 latency on checkout is up,” a trace shows you the one slow database call, three services deep, that’s actually responsible.

Why traces matter

• They show causality, not just correlation. Traces reconstruct the real call graph of a request, which is invaluable once a system has more than a handful of services.
• They pinpoint the bottleneck. Instead of guessing which of ten services is slow, a trace waterfall shows exactly which span dominates the total request time.
• They reveal architecture as it actually is. Auto-generated service maps from trace data are often more accurate than any architecture diagram, since they reflect real traffic.

Key concepts

• Spans and context propagation. Each span has a start time, duration, and metadata; spans are connected across service boundaries by propagating a trace context (typically via HTTP headers) so every hop can attach itself to the same trace.
• Sampling. Capturing every single trace is often too expensive at scale, so systems use head-based sampling (decide upfront) or tail-based sampling (decide after seeing the full trace, so you can keep the interesting ones — errors, outliers).
• Trace context standards. The W3C Trace Context standard defines how trace IDs propagate across services and vendors, which is what allows different tools and languages to interoperate.
• Service maps. Aggregating many traces reveals a dependency graph between services, useful for both architecture understanding and blast-radius analysis during incidents.

Where traces fit with the other pillars

Metrics tell you something is slow; traces tell you exactly where in the call chain. From a specific span, you can usually jump straight to the logs emitted during that request, and if the bottleneck turns out to be CPU-bound code rather than a network call, continuous profiling shows you the exact function responsible.