Distributed Tracing

A span is the fundamental unit of work in a trace, representing a single, named operation within a service. It contains:

• Start and End Timestamps for latency calculation.
• A Span Name (e.g., handle_llm_request).
• A unique Span ID.
• Attributes (key-value pairs) for contextual metadata like model.name="gpt-4" or user.id=12345.
• Span Kind (Client, Server, Producer, Consumer, Internal).
• Status (Ok, Error).
• Links to causally related spans in other traces.

A trace is the complete end-to-end record of a transaction, comprised of a tree of spans. It is defined by:

• A globally unique Trace ID, propagated across all services involved in the request.
• A hierarchical parent-child relationship between spans, showing the flow of execution.
• The Root Span, which has no parent and represents the initial request (e.g., an API call to /chat).
• In LLM serving, a single trace can encapsulate the entire pipeline: user prompt ingestion, retrieval from a vector database, LLM inference, and final response streaming.

Context Propagation is the mechanism that carries the trace context (Trace ID, Span ID, and other metadata) across service and process boundaries. This is essential for correlating spans into a coherent trace.

• W3C TraceContext is the modern standard, using HTTP headers like traceparent and tracestate.
• In asynchronous or message-based systems (e.g., Kafka, RabbitMQ), the context is injected into message headers.
• For LLMs, propagation ensures a user query's trace can follow the request through API gateways, orchestration layers, multiple model calls, and downstream tool executions.

Attributes (or Tags) are key-value pairs attached to spans that provide searchable, filterable context about the operation.

• Examples: llm.model.vendor="openai", retrieval.top_k=5, response.token_count=1500, error.type="rate_limit".

Events (or Annotations) are timestamped logs within a span that mark a specific moment.

• Examples: "cache.hit", "generation.started", "function.called: calculate_risk", "exception thrown" with a stack trace.
• These are critical for debugging the precise timing and cause of issues within a long-running LLM inference.

Instrumentation is the code added to an application to create spans, add attributes, and propagate context. It can be:

• Automatic: Using vendor agents or frameworks (e.g., OpenTelemetry instrumentation for Python's requests library) that inject tracing without code changes.
• Manual: Developers explicitly define span boundaries and attributes in business logic for maximum control and detail.
• For LLM frameworks (LangChain, LlamaIndex), instrumentation captures the execution of chains, agents, and tool calls, turning a black-box workflow into an observable graph.

The Collector is a critical intermediary service that receives, processes, batches, and exports telemetry data (spans) from instrumented applications.

• It can perform filtering, sampling, and data enrichment.
• It exports data to one or more Backends for storage and analysis.

Backend systems are where traces are stored, indexed, and visualized.

• Examples: Jaeger, Zipkin, commercial APM tools (Datadog, New Relic), or specialized LLM observability platforms.
• Here, engineers query traces, analyze service dependencies, create latency histograms, and set alerts based on trace data.

A span is the fundamental unit of work in distributed tracing, representing a single, named operation within a request's lifecycle. It contains timing data, a unique identifier, and metadata about the operation.

• Key attributes: Operation name, start/end timestamps, key-value tags (metadata), and references to parent/child spans.
• Example: A single database query, an HTTP call to another service, or the execution of a specific function within your LLM inference code.
• Spans are nested and linked to form a complete trace, visualizing the entire path of a request.

A trace is a directed acyclic graph (DAG) of spans that records the end-to-end path of a single request as it traverses a distributed system. It provides a holistic view of the request's journey, latency, and dependencies.

• Root Span: The initial span that starts the trace, often at the system's entry point (e.g., an API gateway).
• Causal Relationships: Child spans are causally related to parent spans, showing the flow of execution.
• Primary Use: Diagnosing where latency is introduced (the "critical path") and understanding service dependencies, which is crucial for debugging slow LLM inference chains.

Context propagation is the mechanism that passes trace and span identifiers (collectively called trace context) across service boundaries, enabling the correlation of spans into a single, coherent trace.

• Carriers: The context is typically injected into and extracted from HTTP headers (e.g., traceparent), gRPC metadata, or message queues.
• Requirements: All services in the call chain must be instrumented to participate in propagation.
• Critical for LLMs: Essential for tracing a user query through an LLM application that may call multiple external tools, databases, or other models, maintaining a single request context.

A service mesh is a dedicated infrastructure layer for managing service-to-service communication in a microservices architecture. It provides built-in, transparent capabilities for traffic management, security, and—critically—observability.

• Sidecar Proxy: Services communicate via a lightweight proxy (like Envoy) deployed alongside each service instance.
• Automatic Tracing: The mesh can automatically generate spans for all inter-service traffic without requiring code changes in the application.
• Integration: Often integrates with tracing backends, providing a low-effort way to gain visibility into network-level latency and failures.

Application Performance Monitoring (APM) is a suite of tools and practices for monitoring and managing the performance and availability of software applications. Distributed tracing is a core data source for modern APM platforms.

• Beyond Tracing: APM tools typically aggregate traces with metrics (e.g., error rates, request rates) and logs to provide a unified dashboard.
• Key Features: Automated performance baselining, anomaly detection, alerting, and root cause analysis.
• For LLM Ops: APM is essential for tracking LLM-specific metrics like token generation latency, prompt throughput, and cost-per-request alongside traditional application health signals.