Observability Wiki

Traces

Auto-Instrumentation vs. Manual Instrumentation for Tracing

The trade-off between framework-level tracing that requires no code changes and hand-written spans that capture business-specific context.

Last updated

Getting a service to emit traces requires instrumentation, and there are two fundamentally different ways to add it. Auto-instrumentation (also called zero-code instrumentation) hooks into popular libraries and frameworks automatically — an HTTP server, a database client, a message queue consumer — and generates spans for their calls without the application developer writing any tracing-specific code. Manual instrumentation is the opposite: a developer explicitly creates spans in application code, at exactly the points that matter to them.

Both produce spans that end up in the same trace, but they capture very different things and come with different costs.

Why it matters

  • Auto-instrumentation gets broad coverage almost for free. Adding an agent or an auto-instrumentation package typically requires no source code changes — just a startup flag, an environment variable, or a sidecar — and immediately produces spans for every HTTP request, database query, and outbound call made through a supported library. This is why it’s usually the first thing turned on in a new service.
  • Manual instrumentation is what captures business context auto-instrumentation can’t see. A framework hook knows a database query ran; it has no idea that the query was part of “processing a refund for a premium customer.” Manual spans, with custom names and attributes, are how business-meaningful operations become visible in a trace rather than just technical plumbing.
  • Coverage from auto-instrumentation is limited to what’s supported. Only popular, well-known libraries and frameworks have auto-instrumentation packages; a custom internal library, an unusual protocol, or a niche framework typically isn’t covered, and those gaps show up as blank stretches in an otherwise complete trace.
  • Manual instrumentation has an ongoing maintenance cost auto-instrumentation doesn’t. Every manually created span is code that a developer has to write, review, and keep up to date as the surrounding logic changes, whereas auto-instrumentation updates automatically when the agent or instrumentation package is upgraded.

How they work together

  • Auto-instrumentation typically works through language-specific interception mechanisms. The exact technique varies — bytecode manipulation on the JVM, monkey-patching of modules in Python, runtime hooks in .NET, module-loading hooks in Node.js — but the effect is the same: calls into a supported library are wrapped transparently to create spans without touching the application’s own code.
  • Manual instrumentation uses the tracing API directly. Using an OpenTelemetry SDK (or a vendor SDK), a developer starts a span at the beginning of a logical operation, attaches attributes and events describing what’s happening, and ends the span when the operation completes — giving full control over naming, granularity, and what data gets attached.
  • The two compose naturally within the same trace. Because both approaches ultimately just create spans within the same active trace context, a manually created span for “apply discount” can sit as a parent of auto-instrumented spans for the database calls that operation happens to make — the trace doesn’t distinguish how a span was produced.
  • The common real-world pattern is starting broad, then adding depth. Most teams enable auto-instrumentation first to get baseline visibility across all services with minimal effort, then add manual spans selectively around the specific operations — checkout flows, critical business logic, anything hard to reason about from framework-level spans alone — where the extra detail earns its maintenance cost.

Treating these as an either-or choice usually leaves value on the table in both directions: auto-instrumentation alone tends to produce traces that are technically complete but hard to map back to what the business actually cares about, while manual instrumentation alone is far too labor-intensive to cover an entire system. Using both — automatic hooks for breadth, hand-written spans for the handful of operations that matter most — is what most mature traces setups converge on.

Related tools