Performance Telemetry

Latency measures the time delay for request processing. Key telemetry includes:

• Time to First Token (TTFT): The duration from request submission to the first output token, capturing prompt processing and initial generation overhead.
• Time per Output Token (TPOT): The average time to generate each subsequent token in a stream.
• End-to-End Latency: Total time for a complete request-response cycle.
• Percentile Latency (P50/P90/P99): Statistical distribution showing median (P50) and tail latencies (P90, P99), critical for understanding user-perceived performance.

Throughput measures the processing rate of the system under load. Core telemetry includes:

• Queries Per Second (QPS): The number of complete inference requests processed per second.
• Tokens Per Second (TPS): The total number of output tokens generated across all requests per second, especially relevant for language models.
• Concurrent Request Count: The number of requests being processed simultaneously, which directly impacts both throughput and latency.
• Batch Size: The number of requests grouped for parallel execution, a key lever for GPU utilization.

These metrics quantify how efficiently system hardware is used, identifying bottlenecks.

• GPU Utilization: Percentage of time GPU cores are actively computing, often broken down by streaming multiprocessor (SM) activity.
• GPU Memory Usage: Amount of VRAM allocated and actively used, critical for managing model loading and KV caches.
• CPU Utilization: Load on host processors, which can become a bottleneck for pre/post-processing or scheduling.
• Memory Bandwidth: Rate of data transfer to/from GPU memory, indicating if a workload is memory-bound.
• Power Draw: Watts consumed by accelerators, directly tied to operational cost.

Telemetry that monitors service stability and reliability.

• Request Error Rate: Percentage of requests that fail due to timeouts, model errors, or system faults.
• Hardware Error Counts: ECC memory corrections, thermal throttling events, or PCIe errors.
• Queue Depth: Number of requests waiting to be processed, signaling load imbalance or saturation.
• Service Level Indicators (SLIs): Quantitative measures like latency or availability used to evaluate Service Level Objectives (SLOs).
• Cache Hit/Miss Ratios: For systems using KV caches or other caches, indicating efficiency.

Metrics that provide insight into the internal efficiency of the inference engine and model.

• Tokens per GPU-second: A combined efficiency metric (TPS/GPU) useful for cost analysis.
• FLOPs Utilization: Percentage of the hardware's peak theoretical compute actually achieved.
• KV Cache Usage: Tracking the size and eviction rate of the attention key-value cache.
• Activation Sparsity: For Mixture of Experts (MoE) models, metrics on expert routing and activation.
• Precision Statistics: Monitoring for numerical instability in mixed-precision inference (e.g., FP16, INT8).

Telemetry that translates technical performance into operational and financial terms.

• Cost per Inference: Calculated from resource usage (GPU-seconds) and cloud pricing.
• Inferences per Dollar: A direct efficiency metric for inference cost optimization.
• Saturation Point Identification: The load level where latency degrades sharply, defining operational limits.
• Throughput-Latency Curve Data: The empirical relationship used to provision capacity for target SLOs.
• Model Load Time: Cold start latency, impacting scalability and auto-scaling decisions.

Continuous collection of core metrics provides a live dashboard of inference service health. This involves tracking latency percentiles (P50, P90, P99), throughput (QPS/TPS), and error rates to detect anomalies and ensure compliance with Service Level Objectives (SLOs). Alerts are triggered when metrics breach predefined thresholds, enabling immediate incident response.

• Key Metrics: Request success rate, token generation speed, hardware utilization (GPU/CPU).
• Purpose: Proactive issue detection and maintaining uptime guarantees.

Telemetry enables deep bottleneck analysis by correlating high-level latency with low-level system metrics. Engineers can pinpoint whether a slowdown is due to compute-bound operations (e.g., dense matrix multiplications), memory-bound constraints (e.g., KV cache thrashing), or I/O waits. Tools like performance profilers use this telemetry to visualize execution traces and identify hot spots in model layers or serving infrastructure.

• Key Metrics: Kernel execution time, memory bandwidth usage, cache hit rates.
• Purpose: Targeted optimization of the slowest component in the inference pipeline.

Telemetry links resource consumption directly to business activity. By measuring GPU/CPU utilization, memory footprint, and energy consumption per model, per user, or per API endpoint, organizations can implement precise cost allocation. This data drives inference cost optimization decisions, such as selecting optimal batch sizes, triggering model scaling, or justifying investments in more efficient hardware or model compression techniques like quantization.

• Key Metrics: Cost per 1k tokens, hardware utilization efficiency, power draw.
• Purpose: Financial accountability and justifying infrastructure spend.

Historical and real-time telemetry informs infrastructure scaling decisions. Analyzing throughput-latency curves and identifying the saturation point under different loads allows teams to forecast resource needs. This data is critical for configuring auto-scaling policies in cloud environments, ensuring the system can handle traffic spikes without degrading tail latency or violating SLOs.

• Key Metrics: Requests per second at target latency, concurrent request limits, queue depth.
• Purpose: Right-sizing infrastructure to meet demand predictably and cost-effectively.

When deploying a new model version or optimization (e.g., a quantized variant), telemetry provides the objective data for comparison. Teams run synthetic workloads or shadow real traffic to collect performance baselines for both models. Key comparisons include latency distributions, throughput, and resource utilization. This ensures new deployments meet performance standards and do not introduce a performance regression.

• Key Metrics: Side-by-side P99 latency, throughput differential, memory usage.
• Purpose: Data-driven validation of model and infrastructure changes.

Telemetry translates technical metrics into user-perceived quality. Tracking Time to First Token (TTFT) for interactive applications and Time per Output Token (TPOT) for streaming responses is essential. Monitoring tail latency (P99) ensures that even the slowest requests meet usability standards. This focus on the end-user experience guides optimizations like speculative decoding or improved caching strategies.

• Key Metrics: TTFT, TPOT, P99/P99.9 latency.
• Purpose: Ensuring responsive, predictable applications for all users.

A Service Level Indicator (SLI) is a quantitative, directly measurable metric of a service's performance or reliability. In inference telemetry, common SLIs include:

• Latency percentiles (P50, P90, P99)
• Throughput (Tokens Per Second, Queries Per Second)
• Error rate (percentage of failed requests)
• Availability (uptime percentage)

SLIs are the raw measurements that feed into Service Level Objectives (SLOs). Effective telemetry systems are designed to collect, aggregate, and report these indicators with high fidelity and low overhead.

A Service Level Objective (SLO) is a target value or range for a Service Level Indicator (SLI). It defines the performance threshold a service is expected to meet. For inference systems, SLOs are critical business-engineering contracts.

Examples include:

• P99 latency < 200ms for 99.9% of requests in a calendar month.
• Throughput maintained at > 1000 TPS under defined load.
• Error rate < 0.1%.

Performance telemetry continuously monitors SLIs against SLOs to trigger alerts, guide capacity planning, and validate optimization efforts.

A performance baseline is a set of established reference metrics for an inference system, captured under controlled, reproducible conditions. It serves as the "ground truth" for comparison.

Establishing a baseline involves:

• Running standardized synthetic workloads or real-world workloads.
• Recording key SLIs like latency, throughput, and hardware utilization.
• Documenting system configuration, model version, and infrastructure details.

This baseline is essential for detecting performance regressions after deployments and for quantifying the impact of optimization techniques like model quantization or KV cache management.

A performance profiler is a specialized telemetry tool that collects granular, low-level timing and resource usage data during inference execution. Unlike high-level SLIs, profilers identify the root cause of performance issues.

Key profiling data includes:

• Operator-level execution time (e.g., attention, feed-forward layers).
• GPU kernel duration and memory copy times.
• CPU-GPU synchronization overhead.
• Memory allocation patterns and cache misses.

Tools like NVIDIA Nsight Systems, PyTorch Profiler, and TensorBoard provide visual timelines and bottleneck analysis, guiding optimizations such as operator fusion or memory layout improvements.

Tail latency refers to the high-percentile latencies in a request distribution, typically P95, P99, or P99.9. These represent the slowest requests, which often dictate user-perceived performance.

Causes of high tail latency include:

• Resource contention from concurrent requests.
• Garbage collection pauses.
• Network variability.
• Cold start latency for new model instances.
• Noisy neighbors in shared infrastructure.

Performance telemetry must capture and analyze tail latency separately from median (P50) latency, as optimizing for the median often does not improve the tail. Mitigation strategies involve queue management, load balancing, and over-provisioning.

The throughput-latency curve is a fundamental graph that depicts the relationship between a system's throughput (e.g., Queries Per Second) and its request latency as load increases. It visually defines the operational envelope of an inference service.

Key points on the curve:

• Low load: Latency is stable and minimal; throughput increases linearly.
• Knee of the curve: Latency begins to increase noticeably as the system approaches its saturation point.
• High load/Saturation: Throughput plateaus at its maximum while latency increases sharply due to queuing.

Telemetry systems generate this curve via load testing and stress testing. It is used to determine optimal operating points that balance efficiency (high throughput) with responsiveness (low latency) for given SLOs.