Text Generation Inference (TGI)

Continuous batching (also known as iteration-level batching or incremental batching) is a dynamic scheduling technique that groups incoming inference requests on-the-fly. Unlike static batching, which waits for a fixed batch to fill, TGI's scheduler continuously adds new requests to the current batch as previous ones finish generating tokens. This dramatically improves GPU utilization and throughput, especially under variable request loads.

• Key Benefit: Eliminates idle GPU time by ensuring the compute engine is always working on the next available token for any request.
• Contrast with Static Batching: Static batching suffers from straggler problem, where a single long request delays the entire batch. Continuous batching isolates request completion.

Tensor parallelism is a model parallelism strategy used by TGI to split a single model's layers across multiple GPUs. It partitions individual operations (like matrix multiplications within a transformer layer) so that each GPU holds a portion of the weights and computes a portion of the output. This is essential for serving models larger than the memory of a single GPU.

• How it Works: For a linear layer Y = XW, the weight matrix W is split column-wise. Each GPU computes a partial result, and outputs are aggregated via communication (e.g., all-reduce).
• Communication Overhead: Requires high-speed interconnects (NVLink, InfiniBand) as communication happens during every forward pass.
• Use Case: Serving models like Llama 2 70B or Mixtral 8x7B on a multi-GPU server.

TGI integrates the vLLM engine's PagedAttention algorithm for optimal KV Cache management. The KV cache stores pre-computed key and value states for previous tokens to avoid redundant computation during autoregressive generation. PagedAttention treats this cache like virtual memory, using blocks that can be non-contiguously allocated.

• Solves Fragmentation: Traditional monolithic KV cache allocation leads to memory fragmentation as sequences of different lengths finish, wasting GPU memory. PagedAttention allows efficient reuse of freed blocks.

• Key Impact: Enables higher batch sizes and longer context windows for the same GPU memory, directly increasing throughput. This is a foundational optimization for cost-effective serving.

Token streaming delivers generated text to the client incrementally, as each token is produced, rather than waiting for the entire sequence to complete. TGI implements this using Server-Sent Events (SSE), a standard HTTP protocol for one-way, text-based event streaming from server to client.

• User Experience Benefit: Provides perceived low latency; users see the first word in milliseconds, improving interactivity for chat applications.
• Technical Implementation: The server holds the HTTP connection open and sends a series of data: events containing new tokens. The client (e.g., a browser) uses the EventSource API to listen.
• Contrast with Non-Streaming: Non-streaming (buffered) inference has a long time-to-first-token (TTFT) equal to the full generation time.

TGI uses custom, fused CUDA kernels written in Triton or directly in CUDA to accelerate core transformer operations. These kernels combine multiple sequential operations into a single, optimized GPU kernel, reducing memory reads/writes and leveraging hardware-specific features.

• Common Fusions: FlashAttention for efficient attention computation, layer norm fusions, and activation function fusions (e.g., SwiGLU).
• Benefit: Reduces kernel launch overhead and improves arithmetic intensity, leading to lower latency per token.
• Framework Support: While TGI is built for PyTorch models, these kernels are often framework-agnostic and target the underlying GPU compute primitives.

An inference optimization technique where incoming requests are dynamically grouped and processed together in a single forward pass. Unlike static batching, which waits for a fixed batch size, continuous batching adds new requests to the batch as soon as previous ones finish, maximizing GPU utilization and throughput. This is a core feature of TGI and vLLM, allowing a single server to handle many concurrent users efficiently.

A family of techniques to split a single model across multiple GPUs because it is too large to fit on one device. The two main types are:

• Tensor Parallelism: Splits individual layers (tensors) across devices, requiring frequent communication.
• Pipeline Parallelism: Places different layers (stages) on different devices, passing activations sequentially. TGI natively supports tensor parallelism to serve models like Llama 70B across multiple GPUs, a critical capability for very large models.

A model compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit or 4-bit integers). This drastically reduces the model's memory footprint and can accelerate inference. TGI supports several quantization formats, allowing a 70B parameter model to run on a single GPU. Post-Training Quantization (PTQ) is commonly used, where a pre-trained model is quantized without retraining.