Cold Start Latency

This is the initial phase where the model's weights, architecture, and metadata are read from persistent storage (e.g., disk, network storage, object store) and loaded into the host system's RAM. The latency here is dominated by I/O bandwidth and the size of the model checkpoint.

• Checkpoint Format: Formats like Safetensors load faster than traditional PyTorch .pt files due to security and efficiency optimizations.
• Sharded Models: Large models split across multiple files can load in parallel, reducing time.
• Example: Loading a 70B parameter model (≈140GB in FP16) from a network-attached storage with 1 Gbps bandwidth can take over 15 minutes for the data transfer alone.

Once in memory, the model's computational graph must be optimized for the target hardware (e.g., NVIDIA GPU, AWS Inferentia, Google TPU). This involves:

• Graph Compilation: Frameworks like TensorRT, OpenVINO, or XLA compile the model into a highly optimized, hardware-specific execution plan.
• Kernel Fusion: Combining multiple operations (ops) into a single fused kernel to reduce launch overhead and improve memory locality.
• Just-In-Time (JIT) Compilation: This can be a major contributor to cold start, as the compiler must analyze and optimize the model graph on the first invocation. Subsequent runs use the cached compilation result.

This component involves allocating the necessary GPU memory (or other accelerator memory) and transferring the model weights from host RAM to the device. This is a critical bottleneck.

• Peak Memory Pressure: The system must allocate memory for the model parameters, optimizer states (if training), KV Cache, and activation tensors simultaneously.
• PCIe Bandwidth: Transfer speed is limited by the PCIe bus (e.g., PCIe 4.0 x16 offers ~32 GB/s). A 10B parameter model (20GB) takes >600ms to transfer, assuming ideal conditions.
• Unified Memory: Systems with NVLink or unified CPU-GPU memory (e.g., Apple M-series, AMD APUs) can mitigate this transfer cost.

Before the model can run, the underlying inference runtime and its dependencies must initialize. This includes:

• Starting the Runtime Engine: e.g., initializing TensorRT, ONNX Runtime, or PyTorch's CUDA context.
• Loading Dependencies: Loading shared libraries (.so/.dll files) for operators, communication libraries (e.g., NCCL for multi-GPU), and drivers.
• Warm-up Steps: Some frameworks execute a few dummy inferences to trigger lazy initialization, autotune cuDNN algorithms, or establish CUDA streams. This is often included in cold start measurements.

In cloud/serverless environments (e.g., AWS Lambda, Google Cloud Run, Azure Container Instances), cold start includes the time to provision and start the execution environment itself.

• Container Pull: Downloading the container image from a registry.
• Sandbox Initialization: Starting the microVM or sandbox (e.g., Firecracker).
• Language Runtime Start: For Python-based services, this includes starting the Python interpreter, which has its own overhead.
• Example: A serverless function with a large container image (>1GB) can experience 10-30 seconds of cold start purely from environment provisioning, dwarfing the model loading time.

The final phase involves executing initial requests that prime the system's caches and reach steady-state performance. This is sometimes distinguished as "warm start" latency.

• KV Cache Initialization: For autoregressive models, the Key-Value (KV) cache for the context window is allocated and populated.
• CPU Cache Warm-up: Frequently accessed data (e.g., model weights for early layers) moves into faster CPU caches (L1/L2/L3).
• Auto-tuner Stabilization: Hardware-specific autotuners (e.g., for cuDNN convolution algorithms) may run their final tuning passes.
• JIT Cache Finalization: Remaining just-in-time compiled kernels are finalized and cached for subsequent requests.

Accelerates the disk-to-memory transfer and deserialization of the model, which is a major contributor to cold start time.

• Compressed Formats: Use frameworks like ONNX or TensorRT that store models in optimized, runtime-ready formats.
• Lazy Loading: Load only essential parts of the model initially, fetching larger components (e.g., certain layers) on-demand.
• Sharded Checkpoints: Split large model files into smaller shards for parallel loading.
• Memory-Mapped Weights: Map model weights directly from disk to virtual memory, reducing initial RAM footprint.

Reduces or eliminates the time spent compiling model operations for the specific hardware during the first inference.

• Ahead-of-Time (AOT) Compilation: Pre-compile the model for the target hardware (e.g., using TVM, XLA, or TorchScript) and serve the compiled artifact.
• Kernel Caching: Persist compiled GPU/CPU kernels to disk so they can be reloaded instantly on subsequent cold starts.
• Framework Tuning: Disable runtime optimizations that add overhead for small batch sizes or single requests.

Strategically staggers the initialization process to get the service partially operational faster.

• Two-Phase Startup: First, load a lightweight router/embedding model to accept requests and queue them. Second, load the heavyweight generation model in the background.
• Model Segmentation: For Mixture of Experts (MoE) models, load the shared parameters and a base set of experts first, loading additional experts on-demand based on request routing.
• Feature Extraction First: In multi-stage pipelines, prioritize loading and warming the feature extraction component.

Latency is the total time delay between submitting an inference request and receiving the complete response, measured end-to-end in milliseconds. It is the primary user-facing performance metric.

• Components: Includes network transmission, queuing, compute (cold/warm), and result serialization.
• Measurement: Often reported as an average, but more critically as percentile latency (P50, P90, P99) to understand tail behavior.
• Impact: Directly affects user experience in interactive applications like chatbots and real-time analytics.

Throughput measures the rate at which a system processes work, typically in Queries Per Second (QPS) or Tokens Per Second (TPS). It represents the system's capacity or bandwidth.

• QPS: Counts complete requests, useful for fixed-output tasks (e.g., classification).
• TPS: Counts output tokens, essential for variable-length generative tasks (e.g., LLMs).
• Trade-off: There is a fundamental throughput-latency trade-off; increasing load to maximize throughput typically increases average and tail latency.

Time to First Token is the latency from request submission until the first output token is generated in an autoregressive model (e.g., an LLM). It measures the initial processing overhead.

• Governed by: Prompt encoding, model loading (if cold), and computation of the first output logits.
• Critical for: Perceived responsiveness in streaming applications. Users tolerate a higher TTFT if tokens stream smoothly afterward.
• Relation to Cold Start: A cold start scenario directly and significantly increases TTFT due to model loading and compilation.

Time per Output Token is the average latency to generate each subsequent token after the first in an autoregressive model. It measures the incremental generation cost.

• Governed by: The forward pass of the model for each new token, heavily influenced by KV cache efficiency and memory bandwidth.
• Steady-State Metric: TPOT typically stabilizes after the first token, representing warm inference performance.
• Throughput Link: Aggregate TPOT across all concurrent requests determines the system's maximum Tokens Per Second (TPS).

Tail latency refers to the high-percentile latencies in a request distribution, such as P95, P99, or P99.9. These represent the slowest requests that often dictate user-perceived performance.

• Causes: Can be triggered by resource contention, garbage collection, host OS scheduling, multi-tenant noise, and cold starts.
• Importance: A few slow requests can ruin the experience for many users. Service Level Objectives (SLOs) are often defined around tail latency (e.g., P99 < 200ms).
• Mitigation: Techniques like request hedging, optimized scheduling, and pre-warming aim to reduce tail latency.

Steady-state performance describes the consistent latency and throughput of a system after it has completed initialization phases and resource allocation has stabilized.

• Characteristics: Caches (e.g., KV Cache, model weights) are warm, kernels are compiled, and memory is allocated. TPOT is stable.
• Contrast with Cold Start: The performance plateau achieved after overcoming the initial penalty of cold start latency.
• Benchmarking Target: Most performance benchmarks and Service Level Indicators (SLIs) are measured during steady-state operation to ensure reproducible comparisons.