Distributed Tracing

A trace is the complete record of the lifecycle of a single request, often called a transaction. It is composed of a tree of spans and is identified by a globally unique Trace ID. The trace provides the end-to-end view, showing how a request propagates through services, where time is spent, and where failures occur.

A span represents a single, named, and timed operation within a trace, such as a service call or a database query. It is the fundamental building block. Each span contains:

• A Span ID for unique identification.
• A parent Span ID to establish the tree structure.
• A name describing the operation (e.g., HTTP GET /api/user).
• Timestamps for start and finish.
• Key-value tags for contextual metadata (e.g., http.status_code=200).
• Logs as timestamped events with additional payload.

Context Propagation is the mechanism that carries the tracing context (Trace ID, Span ID, and other metadata) across service boundaries. This is what links spans from different processes into a single trace. Common propagation formats include:

• W3C TraceContext: A modern standard using HTTP headers like traceparent.
• B3 Propagation: Used by Zipkin, with headers like X-B3-TraceId.
• Jaeger Headers: Such as uber-trace-id. Without proper propagation, traces become fragmented and lose their distributed context.

Tags (or Attributes) are key-value pairs attached to a span that describe the context of the operation, such as http.method="GET", db.instance="users", or error=true. They are used for indexing and filtering traces.

Logs (or Events) are timestamped records of discrete events within a span's lifetime, such as an exception stack trace, a debug message, or a note that a cache was missed. While tags describe the span, logs capture events that happened during it.

Sampling is the process of deciding which traces to record and export, as capturing 100% of traces in high-volume systems is often prohibitively expensive. Common strategies include:

• Head-based Sampling: The decision is made at the start of the trace (e.g., record 1% of all traces).
• Tail-based Sampling: The decision is deferred until the end of the trace, based on its outcome (e.g., sample all traces with errors, but only 0.1% of successful ones). Sampling is crucial for controlling cost while retaining visibility into important or anomalous requests.

Instrumentation is the code added to an application to create spans, propagate context, and export trace data. It can be implemented in several ways:

• Manual Instrumentation: Developers explicitly add tracing code to their business logic.
• Automatic Instrumentation: Using agents, SDKs, or frameworks that inject tracing into common libraries (e.g., HTTP clients, database drivers, gRPC) without code changes.
• Hybrid Approaches: Combining automatic instrumentation for common operations with manual instrumentation for critical business logic. Effective instrumentation is key to achieving comprehensive observability.

A trace is the complete end-to-end journey of a single request. It is composed of multiple spans, which represent individual units of work (e.g., a model inference, a database call). Context propagation is the mechanism (using headers like traceparent) that passes the trace identifier across service boundaries, network calls, and asynchronous processes, ensuring all related spans are linked.

• Span Attributes: Key-value pairs recording metadata like model.name, input.tokens, peer.service.
• Span Events & Status: Timestamped logs for errors or milestones, and a final status code (OK, ERROR).

For a Retrieval-Augmented Generation (RAG) request, a trace visualizes the entire workflow:

• Root Span: User query received by API gateway.
• Child Span 1: Query embedding generation.
• Child Span 2: Vector database similarity search.
• Child Span 3: Context formatting and prompt construction.
• Child Span 4: LLM inference generation (with sub-spans for prefill, decoding).
• Child Span 5: Post-processing and response formatting.

This reveals bottlenecks—whether latency is in retrieval, token generation, or a specific microservice.

In production PEFT servers, tracing is essential for monitoring adapter switching and multi-adapter serving. Key spans track:

• Adapter Load Latency: Time to fetch and activate a specific LoRA or adapter module.
• Cache Hit/Miss: For cached adapter weights.
• Tenant Isolation: Attributing resource usage (GPU time, memory) to specific tenants or tasks via span attributes (tenant.id, adapter.id).

This data validates performance isolation and helps debug issues where the wrong adapter is selected or loading is slow.

Traces transform debugging from guesswork to root-cause analysis. Engineers can:

• Identify High Latency Spans: Pinpoint if delay is in pre-processing, a specific model, or network calls between services.
• Detect Cascading Failures: See how an error in a vector database call propagates to cause a failure in the LLM span.
• Analyze Performance Regressions: Compare trace durations and resource attributes (e.g., gpu.utilization) before and after a model or infrastructure deployment.
• Correlate with Metrics & Logs: Use the trace ID to find related high-cardinality metrics (prompt tokens/sec) and detailed error logs.

Effective tracing requires deliberate design:

• Meaningful Span Names: Use verb-noun pairs like embedding.generate, lora.adapter.load.
• Structured Attributes: Use consistent key naming (e.g., llm.model.vendor, retriever.top_k) for aggregation.
• Sample Intelligently: Use head-based sampling for low-volume, high-value requests and tail-based sampling for errors or high-latency traces.
• Trace LLM Vendor Calls: Instrument calls to external APIs (OpenAI, Anthropic) to attribute cost and latency to user requests.
• Link to Business Metrics: Enrich spans with business context like user.session.id or inference.cost.estimated.

Observability is a measure of how well the internal states of a system can be inferred from its external outputs. It is the superset discipline that encompasses distributed tracing, along with metrics and logs, to provide a holistic view of system health and behavior.

• The Three Pillars: Metrics (quantitative measurements), Logs (timestamped event records), and Traces (request lifecycle context) together form the foundation.
• Purpose: Enables engineers to ask arbitrary questions about system performance without pre-instrumenting for those specific questions, moving beyond basic monitoring.
• Critical for AI Serving: In production PEFT servers, observability is required to correlate model latency spikes with specific adapter loads, GPU memory pressure, or data drift in input features.

Telemetry refers to the automated collection and transmission of measurements and operational data from remote sources. It is the raw data fuel for observability.

• Data Sources: Includes application performance metrics, structured logs, span data from traces, and infrastructure health signals.
• Instrumentation: The code added to an application to emit telemetry. In AI servers, this involves hooking into model inference loops, adapter switching logic, and GPU kernel executions.
• Pipeline: Telemetry data flows through agents, collectors (e.g., OpenTelemetry Collector), and into backends like Prometheus for metrics or Jaeger/Tempo for traces.

A Span represents a single, named, and timed operation representing a unit of work within a distributed trace. It is the fundamental building block of a trace.

• Attributes: Contains key-value pairs describing the operation (e.g., http.method=POST, model.adapter_id="lora-finance-01").
• Timing: Has a start and end timestamp, defining its duration.
• Context: Carries a SpanContext, which includes the globally unique Trace ID and the span's own Span ID, enabling correlation.
• Hierarchy: Spans have parent-child relationships, forming a tree structure that models the flow of a request. In an inference request, you might have a root span for the API call, with child spans for tokenization, adapter selection, and layer-by-layer model execution.

A Trace is a visualization of the end-to-end journey of a single request as it propagates through a distributed system. It is a directed acyclic graph (DAG) of spans.

• Trace ID: A unique identifier that is consistent across all spans belonging to the same request, enabling aggregation.
• Purpose: Answers the question, "What happened to this specific request?" It shows the causal and temporal relationships between all operations triggered by that request.
• AI Serving Example: A trace for a /generate request would show the HTTP server span, the routing to a specific model replica, the loading of a LoRA adapter, the GPU inference steps, the token streaming, and any downstream calls to databases or other microservices.

Context Propagation is the mechanism by which trace context (e.g., Trace ID, Span ID, baggage) is passed across service boundaries, ensuring all operations belong to the same logical trace.

• Carriers: Context is injected into and extracted from carriers, which are typically HTTP headers (e.g., traceparent), gRPC metadata, or message queues.
• W3C Trace Context: A standard HTTP header format (traceparent, tracestate) that ensures interoperability between different tracing systems and services.
• Critical for Distributed Systems: Without proper propagation, a trace would break at every network hop. In a microservices architecture for AI, this allows tracing a user query from the API gateway through an orchestration service, to a specific inference server, and back.