Instrumentation

Instrumenting an HTTP server involves creating a span for each incoming request. Key steps include:

• Extracting trace context from incoming headers (e.g., traceparent).
• Creating a server span with attributes like http.method, http.route, and http.status_code.
• Propagating the context to downstream calls (database, other services).
• Recording latency and any errors.

Example: A Python FastAPI endpoint instrumented with OpenTelemetry to trace request duration and status.

This captures the performance of queries and commands sent to databases. Instrumentation typically:

• Wraps the database driver to intercept calls.
• Creates a client span for each query/operation.
• Records attributes such as:
  • db.system (e.g., postgresql, redis)
  • db.statement (often sanitized)
  • db.operation (e.g., SELECT, SET)
• Measures query execution time and connection pool metrics.

Example: Auto-instrumentation for the pg (PostgreSQL) Node.js client that traces all queries.

Instrumenting asynchronous messaging systems is critical for tracing workflows across decoupled services.

For a Producer:

• Inject trace context into the message headers (e.g., Kafka headers, AMQP properties).
• Create a producer span linked to the sending operation.

For a Consumer:

• Extract context from the message headers.
• Create a consumer span representing the processing of the message.
• Link the consumer span to the producer's span to connect the asynchronous workflow.

Example: Instrumenting an Apache Kafka client to trace message publication and consumption latency.

Instrumenting outbound API calls ensures downstream service work is part of the parent trace.

• Inject the current trace context into the outgoing HTTP request headers.
• Create a client span for the external call.
• Record attributes like:
  • http.url
  • http.method
  • peer.service (name of the called service)
• Measure external service latency, which is a major contributor to end-user latency.

Example: Using an instrumented fetch or requests library where trace context is automatically propagated.

For complex business logic within a service, creating internal spans provides granular visibility.

• Manually create spans around significant code blocks or functions.
• Use span attributes to log business context (e.g., user.id, transaction.amount, decision.score).
• Record events within the span to mark specific milestones or state changes.
• Set the span status to error on exceptions.

This is often manual instrumentation and is key for understanding the performance of agentic reasoning loops or planning algorithms.

Example: Wrapping a calculate_risk_score() function or an LLM inference call in a dedicated span.

Auto-instrumentation uses agents, bytecode manipulation, or wrappers to inject tracing without code changes.

• Dynamically injects tracing into common frameworks and libraries (e.g., Express.js, Spring Boot, Django).
• Captures spans for HTTP servers, database clients, and RPC calls automatically.
• Manages context propagation across instrumented calls.

Trade-off: Provides immediate, broad coverage but less customization than manual instrumentation. Essential for quickly enabling observability in large, existing codebases.

Example: The OpenTelemetry Java Agent JAR file attached to a Spring Boot application.

The automatic injection of tracing code into an application at runtime, eliminating the need for manual developer effort. This is typically achieved through language-specific agents or SDKs that hook into frameworks and libraries.

• Mechanism: Uses bytecode manipulation, monkey-patching, or eBPF to wrap key functions like HTTP clients, database drivers, and RPC calls.
• Coverage: Provides out-of-the-box support for common frameworks (e.g., Express, Spring Boot, Django).
• Trade-off: While fast to implement, it may lack deep business context compared to manual instrumentation.

A vendor-neutral, open-source observability framework that provides the APIs, SDKs, and tools required to generate, collect, and export telemetry data (traces, metrics, logs). It is the de facto standard for modern instrumentation.

• Components: Consists of a cross-language specification, per-language SDKs, the OTLP protocol, and the Collector.
• Goal: Decouples instrumentation from analysis, allowing data to be sent to any compatible backend (e.g., Jaeger, Prometheus, commercial vendors).
• Adoption: Replaces legacy, vendor-specific SDKs, reducing lock-in and standardizing the instrumentation layer.

The critical mechanism by which trace context (Trace ID, Span ID) is passed between services to maintain the continuity of a trace across process and network boundaries.

• Carriers: Context is propagated via HTTP headers, gRPC metadata, or message queues (e.g., Kafka, RabbitMQ).
• Propagators: Libraries contain injectors (for outbound requests) and extractors (for inbound requests) for specific formats like W3C Trace Context or B3 Propagation.
• Failure Impact: Broken propagation results in fragmented traces, making end-to-end latency analysis impossible.

Key-value pairs attached to a span that provide descriptive, queryable metadata about the operation it represents. They are the primary vehicle for adding business and operational context to telemetry.

• Standard Attributes: Include HTTP status codes, database query strings, and RPC method names.
• Business Attributes: Custom attributes like user.id=abc123 or shopping.cart.value=299.99 transform generic traces into actionable business observability.
• Cardinality Caution: High-cardinality attributes (e.g., full URLs with IDs) can dramatically increase storage costs and require careful schema design.

The process of selectively capturing a subset of traces to manage data volume, storage costs, and processing overhead, which is essential for high-throughput systems.

• Head Sampling: Decision is made at the start of a request (e.g., sample 10% of all traces). Simple but can miss important, rare events.
• Tail Sampling: Decision is made after the trace is complete, based on its full context (e.g., "keep all traces with errors or latency > 2s"). More powerful but requires a buffering component like the OpenTelemetry Collector.
• Adaptive Sampling: Dynamically adjusts sampling rates based on current system load or traffic patterns.

A vendor-agnostic proxy that receives, processes, and exports telemetry data. It acts as a universal adapter and processing hub in an observability pipeline, decoupling instrumentation from backends.

• Components: Built with receivers, processors, and exporters connected via pipelines.
• Key Functions: Can perform batch processing, tail sampling, trace enrichment (adding attributes), and format translation (e.g., OTLP to Jaeger).
• Deployment: Often run as a sidecar or daemonset in Kubernetes to receive traffic from local instrumented applications.