OpenTelemetry Collector

Receivers are how data gets into the Collector. They listen for data in various formats and protocols, acting as the ingestion layer. Key types include:

• OTLP Receiver: The native receiver for the OpenTelemetry Protocol (gRPC/HTTP).
• Push-based Receivers: Accept data sent to them (e.g., Jaeger, Zipkin).
• Pull-based Receivers: Actively scrape for data (e.g., Prometheus, hostmetrics). This design allows a single Collector instance to consolidate data from dozens of heterogeneous sources, simplifying the observability pipeline.

Processors transform, filter, and route telemetry data between receivers and exporters. They are the core of the Collector's data manipulation capabilities. Common processors include:

• Batch Processor: Groups spans and metrics to improve compression and reduce export overhead.
• Attribute Processor: Adds, updates, or deletes span attributes (e.g., adding environment=prod).
• Filter Processor: Drops telemetry based on conditions like span name or error status.
• Tail Sampling Processor: Makes sampling decisions after a trace is complete, based on its full context (e.g., "keep all traces with errors"). Processors are configured in a pipeline, allowing sequential operations like batching → filtering → enrichment.

Exporters are how data leaves the Collector, sending processed telemetry to one or more backends or analysis tools. They handle the final serialization and transmission. Examples include:

• OTLP Exporter: Sends data to any OTLP-compatible backend.
• Vendor-specific Exporters: Send data to commercial observability platforms (e.g., Datadog, New Relic, Dynatrace).
• Logging/ Debug Exporter: Writes data to stdout or a file for local debugging. A single pipeline can fan out to multiple exporters, enabling a multi-vendor strategy or sending copies of data to long-term storage and real-time monitoring simultaneously.

The Collector supports two primary deployment patterns that define its role in the architecture:

• Agent Mode: Deployed as a sidecar or daemonset on each host. Its primary jobs are:
  • Receiving telemetry from local applications.
  • Performing initial processing (e.g., batching, sampling).
  • Relaying data to a central Collector gateway. This offloads processing from the application and provides a local buffer.
• Gateway Mode: Deployed as a centralized service (often in a cluster). It aggregates data from many agents or direct sources, performs heavy processing (e.g., tail sampling, enrichment), and exports to final backends. This separation of concerns is critical for scaling and managing data pipelines.

A core tenet of the OpenTelemetry Collector is its neutrality. It decouples instrumentation from analysis by acting as a universal adapter.

• Protocol Translation: It can receive data in one format (e.g., Jaeger Thrift) and export it in another (e.g., OTLP to a vendor backend).
• Backend Independence: Teams can switch observability backends by changing exporter configuration, without altering application instrumentation.
• Legacy System Integration: It can ingest data from older systems (Zipkin, Jaeger, StatsD) and forward it to modern OTLP-based pipelines. This makes it a future-proof hub, reducing lock-in and simplifying the management of complex, multi-tool observability landscapes.

Collector behavior is defined declaratively via YAML configuration files, which specify pipelines for traces, metrics, and logs. A pipeline links a receiver, a series of processors, and one or more exporters. Example Pipeline (traces):

yamlCopy
receivers: [otlp, jaeger]
processors: [batch, attributes]
exporters: [otlp, logging]

The Collector is also highly extensible. The community and vendors can build:

• Custom Receivers/Exporters for proprietary protocols.
• Custom Processors for unique business logic (e.g., PII redaction).
• Extensions for non-pipeline functionality like health monitoring. This open model allows the Collector to adapt to virtually any enterprise telemetry requirement.

The Collector acts as a single point of entry for all observability signals from an agentic system. It natively supports:

• OTLP (OpenTelemetry Protocol) for traces, metrics, and logs.
• Legacy formats like Jaeger, Zipkin, and Prometheus.
• This allows heterogeneous agent components, written in different languages or using different legacy SDKs, to send data to one location, simplifying instrumentation and reducing vendor lock-in.

For distributed tracing to work across an agent's internal steps and external tool calls, trace context must be preserved. The Collector is critical for:

• Receiving spans with W3C Trace Context headers intact.
• Ensuring Trace IDs and Span IDs are not corrupted during processing.
• Propagating context when the Collector itself makes calls (e.g., for enrichment), maintaining the integrity of the end-to-end trace.
• This is foundational for visualizing the complete agent reasoning traceability graph.

Before export, the Collector can transform telemetry data using configured processors. For agent observability, this enables:

• Trace enrichment: Adding span attributes like agent.session_id, agent.workflow_name, or tool_call.success to all relevant spans.
• Filtering: Dropping noisy or low-value internal operations to reduce cost and focus on business logic.
• Tail sampling: Implementing sampling rules based on the complete trace, such as "always sample traces where http.status_code = 500" or "sample 100% of traces for a specific high-value agent."
• Redaction: Removing sensitive data (e.g., PII from prompts) from spans and logs.

The Collector decouples data production from consumption. It can route processed telemetry to multiple backends simultaneously via exporters, which is essential for:

• Sending traces to a distributed tracing backend like Jaeger or Tempo for latency analysis.
• Sending derived metrics to Prometheus or a commercial APM for agent performance benchmarking.
• Sending logs to Elasticsearch or Loki for agent behavior auditing.
• Duplicating data to a low-cost storage for long-term compliance archives.
• This supports a polyglot observability strategy without burdening the agent runtime.

Deployed as a sidecar or daemonset, the Collector provides a buffer between agents and observability backends, enhancing system resilience:

• Batching and retries: Aggregates data and retries failed exports, preventing data loss during backend outages.
• Load shedding: Can apply rate limiting or sampling under high load to protect backends.
• Network optimization: Reduces the number of persistent connections from many agents to a few Collectors.
• This is critical for maintaining agentic SLI/SLO definitions, as observability failure should not impact agent execution.

By processing all agent telemetry, the Collector enables the construction of higher-level, agent-specific observability constructs:

• Service graphs can be filtered to show only services involved in agent workflows.
• Trace correlation is simplified, allowing logs and metrics from an agent's tool calls to be linked to its root trace.
• Custom metrics can be derived from trace data (e.g., agent.planning_duration calculated from span timings) and exported.
• This centralized processing is a prerequisite for effective multi-agent observability and agentic anomaly detection systems.

OTLP (OpenTelemetry Protocol) is the primary, vendor-agnostic wire protocol for transmitting telemetry data. It is the recommended method for sending data to an OpenTelemetry Collector. Characteristics:

• Supports gRPC and HTTP/JSON transports for flexibility in different network environments.
• Designed for efficiency with protocol buffers for serialization.
• The Collector uses OTLP as its default receiving and exporting format, but can convert to/from other formats like Jaeger or Zipkin.

Distributed tracing is the methodology of tracking a request's journey across service boundaries. The Collector is a central hub for processing these traces. Core concepts it handles:

• Trace: The end-to-end record of a request, composed of many spans.
• Context Propagation: The mechanism (via headers like W3C TraceContext) that passes trace IDs between services, which the Collector can validate and forward.
• The Collector aggregates spans from multiple services to reconstruct complete traces for analysis.

A span represents a single, named, timed operation within a trace (e.g., a function call, database query). The Collector processes millions of spans. Key span properties the Collector can manipulate:

• Attributes: Key-value pairs (e.g., http.method=GET) that the Collector can filter or enrich.
• Span Kind: Classifies the role (Client, Server, Internal, etc.), which can inform processing logic.
• Span Links & Events: References to other traces or timed annotations within a span.

Trace sampling is the critical process of reducing telemetry volume by selectively capturing traces. The Collector is where sophisticated sampling policies are often enforced.

• Head Sampling: Decision made at the start of a request (e.g., sample 10% of traces). Can be done by the Collector if it receives unsampled data.
• Tail Sampling: Decision made after a trace is complete, based on its full context. This is a powerful feature of the Collector, allowing rules like "sample all traces with errors" or "sample traces slower than 2 seconds," which requires seeing the entire trace.

A service graph is a topological map showing dependencies between services, automatically generated from trace data. The Collector can be configured to generate service graph metrics.

• It analyzes span.kind (Client/Server) and peer.service attributes to infer relationships.
• Outputs metrics like request counts and error rates between service pairs, which can be exported to monitoring backends.
• This provides a real-time, data-driven view of system architecture and dependency health.