TensorRT-LLM

TensorRT-LLM performs aggressive kernel fusion, combining multiple GPU operations into single, custom CUDA kernels. This reduces the overhead of launching numerous small kernels and minimizes data transfers between GPU memory hierarchies.

• Fused Multi-Head Attention (FMHA): Combines the entire attention computation (QKV projection, attention scoring, softmax, output projection) into one optimized kernel.
• LayerNorm/Activation Fusion: Merges normalization layers with subsequent activation functions (e.g., GeLU, SiLU).
• Result: Significantly reduces launch overhead and improves instruction-level parallelism, directly increasing tokens/sec/watt.

TensorRT-LLM implements continuous (in-flight) batching to dynamically group requests, allowing new sequences to join and finished ones to exit the batch without stalling the GPU. This is powered by its integration of the PagedAttention algorithm for KV cache management.

• Non-Contiguous Memory Pages: The KV cache is split into fixed-size blocks, enabling efficient sharing of GPU memory across sequences.
• Eliminates Internal Fragmentation: Unlike monolithic caches, paging allows memory from completed sequences to be instantly reused, achieving near 100% memory utilization.
• Enables Variable-Length Batching: Efficiently handles requests with vastly different context lengths in the same batch.

The SDK provides extensive support for post-training quantization (PTQ) and quantization-aware training (QAT) to reduce model footprint and accelerate computation.

• Weight-Only & Activation Quantization: Converts FP16/BF16 weights to INT8 or FP8, often with per-channel scaling for minimal accuracy loss.
• KV Cache Quantization: Stores key and value tensors in INT8 or FP8, dramatically reducing the memory bandwidth required during the decode phase.
• Automatic Precision Calibration: Uses a representative dataset to determine optimal scaling factors, ensuring model quality is preserved post-quantization.

At its core, TensorRT-LLM is a just-in-time (JIT) compiler. It takes a model definition (from PyTorch, TensorFlow, or ONNX) and applies a series of graph-level optimizations before generating highly optimized engine files (.engine).

• Constant Folding: Pre-computes operations on constant tensors.
• Operator Vertical/Horizontal Fusion: Merges layers across the computational graph.
• Pattern Replacement: Swaps subgraphs for more hardware-efficient implementations.
• Target-Specific Kernels: Generates different code paths for different NVIDIA GPU architectures (e.g., Hopper, Ada Lovelace).

For models that exceed the memory of a single GPU, TensorRT-LLM provides native support for model parallelism strategies, enabling inference on multi-GPU systems.

• Tensor Parallelism (Intra-Layer): Splits individual model layers (e.g., the weight matrices of an MLP) across multiple GPUs, requiring all-to-all communication during forward passes.
• Pipeline Parallelism (Inter-Layer): Places different groups of model layers on different GPUs, using a micro-batching approach to keep all devices utilized.
• Optimized Communication: Uses NVIDIA NCCL for high-speed inter-GPU data transfers, minimizing the overhead of parallelism.

TensorRT-LLM includes native support for speculative decoding, a technique to reduce inference latency by using a smaller, faster draft model to propose tokens that are then verified in parallel by the larger target model.

• Draft & Target Model Execution: Manages the concurrent execution of two separate model instances on the same hardware.
• Efficient Verification Kernel: Implements an optimized kernel for the token acceptance/rejection step, which compares the draft and target model outputs.
• Performance Gain: Can achieve 2-3x lower latency for the same model by reducing the number of serial decoding steps from the large model.

Continuous batching (or iterative batching) is a dynamic scheduling paradigm that TensorRT-LLM implements to maximize GPU utilization. Unlike static batching, it allows new requests to join and completed sequences to exit a running batch independently.

• Maximizes Throughput: By keeping the GPU constantly saturated with work, it significantly improves aggregate tokens/sec compared to static batch processing.
• Reduces Latency: Users experience lower time-to-first-token (TTFT) as their request doesn't wait for a fixed batch interval to start.
• Requires Advanced KV Cache Management: This technique is only possible with a memory manager like PagedAttention, which can dynamically allocate and deallocate cache pages for sequences entering and leaving the batch.

In-flight batching is TensorRT-LLM's specific, advanced implementation of continuous batching that provides fine-grained control over the execution graph. It treats requests as individual data streams that are dynamically woven together for execution.

• Context-Level Parallelism: Enables the prefill phase (processing the initial prompt) of one request to execute concurrently with the decode phase (token generation) of another, hiding latency.
• Optimized for Variable Input/Output: Efficiently handles the highly heterogeneous workloads typical of chat-based inference, where prompts and generated lengths vary widely.
• Core to TensorRT-LLM's Performance: This scheduler, combined with fused kernels, is a key differentiator for achieving high throughput on NVIDIA GPUs.

KV cache quantization is a memory compression technique supported by TensorRT-LLM where the key and value tensors are stored in a lower numerical precision format (e.g., FP8, INT8) instead of FP16 or BF16.

• Reduces Memory Bandwidth Pressure: Loading lower-precision cache entries from GPU memory is faster, alleviating a major bottleneck in the memory-bound decode phase.
• Increases Effective Cache Capacity: Allows for longer context windows or larger batch sizes within the same GPU memory footprint.
• Quality-Aware Algorithms: TensorRT-LLM employs advanced quantization algorithms, often using FP8, which typically has a negligible impact on output quality compared to more aggressive INT8 quantization of the cache.

Kernel fusion is a compiler-level optimization where multiple sequential GPU operations (kernels) are combined into a single, custom kernel. This is a core strength of the TensorRT compiler underlying TensorRT-LLM.

• Reduces Kernel Launch Overhead: Eliminates the latency of launching dozens of small, separate kernels for each layer of a transformer.
• Improves Memory Locality: Intermediate results are kept in fast registers or shared memory instead of being written to and read from slow global GPU memory (HBM).
• Enables Novel Optimizations: Allows for the creation of fused attention kernels that integrate operations like RoPE (Rotary Positional Embedding), attention masking, and softmax into one efficient pass.

Beyond the KV cache, TensorRT-LLM provides comprehensive support for compressing the model weights themselves, a prerequisite for high-performance inference.

• Post-Training Quantization (PTQ): Converts model weights to lower precision (e.g., INT8, FP8) after training with minimal calibration data. TensorRT-LLM includes advanced calibration algorithms.
• Weight Sparsity: Supports 2:4 structured sparsity patterns, where two of every four weights are pruned to zero. NVIDIA GPUs from the Ampere architecture onward have dedicated hardware (SPARSITY cores) to accelerate computation on these sparse matrices, doubling effective throughput.
• Quantization-Aware Training (QAT): Framework for training models that will be deployed with quantization, often yielding higher accuracy than PTQ.