OpenTelemetry Collector Architecture and Pipelines

The OpenTelemetry Collector is a standalone, vendor-neutral service that sits between instrumented applications and observability backends. It receives telemetry, transforms it, and forwards it on — so applications only need to know how to speak OTLP to a nearby Collector, and the Collector handles routing, filtering, and fan-out to whatever backends an organization actually uses.

Internally, the Collector is built around a strict pipeline model: receivers accept data in, processors transform it in flight, and exporters send it out. Understanding this model is the difference between a Collector config that quietly drops data under load and one that survives production traffic.

Why it matters

• Decouples instrumentation from backends. Applications emit OTLP once; the Collector config — not application code — decides where telemetry ends up. Switching or adding a backend is a config change, not a redeploy.
• Centralizes processing that shouldn’t live in every service. Batching, sampling, redacting sensitive attributes, and enriching data with resource metadata are done once in the Collector instead of duplicated across every language and service.
• Protects backends and the network from bursty telemetry. Batching and memory limiting smooth out spiky data volumes before they hit an ingest API or saturate a link.
• Enables fan-out without touching application code. A single pipeline can export the same data to multiple destinations simultaneously — useful for running two backends in parallel during a migration or evaluation.

How it works

• Pipelines are per-signal. A Collector config defines separate pipelines for traces, metrics, and logs (service.pipelines.traces, .metrics, .logs), each wiring together its own list of receivers, processors, and exporters. The same receiver or exporter can be referenced by multiple pipelines, but data flows through each pipeline independently.
• Data flows receiver to processor to processor to exporter, in order. Everything a receiver accepts is pushed to the first processor in the pipeline, which does its work and pushes to the next, and so on until the last processor hands off to every exporter configured for that pipeline. A processor can also drop data outright — that’s how filtering and sampling processors work.
• memory_limiter should be the first processor in every pipeline. It applies backpressure and starts refusing data before the Collector’s memory usage causes an OOM kill. Without it, a traffic spike or a backend outage causes unbounded in-memory buffering and the Collector eventually crashes, losing everything it was holding.
• batch groups telemetry before export. Sending one network call per span or metric point is wasteful; batching amortizes that cost and is close to mandatory in any real deployment.
• tail_sampling makes sampling decisions after seeing a whole trace. Unlike head sampling (decided at the start of a trace, usually in the SDK), tail sampling can keep traces because they were slow or errored, and drop uninteresting ones — but it requires every span of a given trace to arrive at the same Collector instance, which shapes how you deploy it.
• attributes and resource processors add, remove, or rewrite metadata — for example scrubbing a field known to hold PII, or attaching a k8s.pod.name resource attribute picked up from the environment.

Agent vs. gateway deployment

The Collector can be deployed in two complementary roles, and most production setups use both:

• Agent mode runs a Collector instance close to the workload — as a sidecar in the same pod, or a DaemonSet with one instance per node. Agents do lightweight work: receiving OTLP from local processes, adding local resource attributes, and forwarding onward. Because an agent sees only a fraction of the fleet’s traffic, it can’t do trace-complete operations like tail sampling.
• Gateway mode runs a smaller, centralized, horizontally scalable pool of Collectors (typically a Deployment behind a load balancer) that agents forward to. Gateways are where heavyweight, fleet-wide processing happens — tail sampling, complex routing, and the final export to backends — because a properly configured gateway tier (often using a load-balancing exporter keyed on trace ID) guarantees that all spans belonging to one trace land on the same gateway instance.

This agent-to-gateway pattern keeps per-node resource usage low while concentrating expensive, stateful processing where it can actually see the complete picture. It also gives a single place — the gateway — to change or add export destinations without redeploying anything running alongside application code. Backends like Jaeger, Grafana Tempo, and Prometheus are commonly plugged in at the exporter stage of this gateway tier. See the OpenTelemetry hub overview for how the Collector fits alongside the SDKs and semantic conventions.