Performance Baseline

A baseline must capture the full latency distribution, not just averages. Key metrics include:

• End-to-End Latency: The total time from client request to complete response receipt, including network and processing.
• Tail Latency (P95/P99): The high-percentile response times critical for understanding worst-case user experience and system stability.
• Time to First Token (TTFT): For streaming applications, the delay until the first output token is generated, defining perceived responsiveness.
• Time Per Output Token (TPOT): The average latency for each subsequent token, governing streaming speed.

Latency is meaningless without specifying the concurrent load. The baseline defines the system's capacity profile:

• Queries Per Second (QPS): The request throughput the system can sustain.
• Concurrent Requests: The number of simultaneous inference queries being processed.
• Throughput-Latency Curve: A graph plotting latency against increasing QPS, identifying the performance knee where latency degrades exponentially. The optimal operating point is typically just before this knee.

The baseline is intrinsically tied to a precise, versioned snapshot of the entire serving stack. This includes:

• Model Version & Precision: e.g., Llama-3-70B-Instruct-FP8.
• Inference Engine & Version: e.g., vLLM 0.4.2 with specific configuration flags.
• Hardware Specification: GPU type (e.g., H100 80GB PCIe), CPU, memory, and interconnect details.
• Serving Configuration: Batch size, scheduling policy (e.g., continuous batching), and KV cache parameters.
• Infrastructure: Container image, OS kernel version, and driver versions.

The baseline must be established using a synthetic but representative dataset that mirrors production traffic in key aspects:

• Input/Output Payload Size Distribution: Mimicking real-world prompt and completion lengths.
• Request Arrival Pattern: Simulating real traffic bursts or steady-state load.
• Query Mix: If applicable, representing different types of inference tasks (e.g., chat, summarization, classification). Using an unrealistic, trivial workload (e.g., all 10-token prompts) creates a useless baseline that won't detect real-world regressions.

A baseline is a statistical measurement requiring controlled conditions and sufficient data:

• Warm State Measurement: Metrics are captured after the cold start latency period, with models loaded and caches warmed.
• Measurement Duration: A sustained run (e.g., 10-30 minutes) to account for variability and capture steady-state performance.
• Elimination of External Noise: Runs should be on dedicated, non-contended hardware to isolate system performance.
• Clear Percentiles and Confidence Intervals: Reporting not just averages but distributions with measured variance (e.g., P50, P95, P99 latency ± 5ms).

The performance baseline directly informs and validates latency SLOs. A complete baseline includes the verified SLO targets it supports, such as:

• Primary SLO: P99 end-to-end latency < 2.0 seconds at 100 QPS.
• Secondary SLOs: P95 TTFT < 500ms, Average TPOT < 75ms. These SLOs, derived from the baseline, become the contractual performance goals for the service, used to manage error budgets and trigger rollbacks during canary analysis.

The total time delay between submitting an input to a machine learning model and receiving its corresponding output. This is the core measurement for which a performance baseline is established.

• Encompasses: Model computation, data transfer, and any preprocessing/postprocessing.
• Baseline Use: Serves as the primary metric to track for regression detection after model or infrastructure changes.

The high-percentile response times (e.g., the 95th or 99th percentile) that represent the slowest requests in a distribution. While a baseline tracks average latency, tail latency baselines are critical for SLOs and user experience guarantees.

• Indicates System Stability: Spikes in P99 latency often reveal resource contention, garbage collection, or queuing issues not visible in averages.
• Baseline Comparison: Establishing a P99 baseline allows teams to set and enforce Service Level Objectives (SLOs).

A target reliability goal defined for a specific latency percentile, forming the basis for performance agreements. A performance baseline is the empirical data used to define and validate a realistic SLO.

• Example SLO: "P99 inference latency < 300ms."
• Relationship to Baseline: The baseline measurement under expected load informs what SLO is achievable. Deviations from the baseline consume the error budget.

A graph plotting the relationship between a system's request throughput (Queries Per Second) and its corresponding average or tail latency. The performance baseline is a single point on this curve under defined load.

• Identifies Optimal Operating Point: Shows where adding more concurrent requests causes latency to degrade non-linearly.
• Baseline Context: A baseline is invalidated if throughput changes significantly, as latency is load-dependent.

A deployment strategy where a new model or configuration is released to a small subset of production traffic. Its performance is compared against the established performance baseline from the stable version.

• Process: Metrics (latency, error rate) from the canary group are statistically compared to the baseline group.
• Purpose: To detect regressions before a full rollout, using the baseline as the control.

The process of using profiling and metrics to pinpoint the specific system component limiting performance. A performance baseline provides the "normal" profile from which deviations indicate a new bottleneck.

• Tools: CPU/GPU profilers (PyTorch Profiler, NVIDIA Nsight), tracing, and system metrics.
• Baseline Comparison: A slowdown against the baseline directs profiling efforts to specific subsystems (e.g., GPU kernels, data loading, network).