TensorRT

TensorRT performs graph optimization by fusing multiple network layers and operations into a single, larger kernel. This reduces the number of kernel launches and memory transfers, which are significant overheads in a naive layer-by-layer execution.

• Example: A convolution, bias addition, and ReLU activation can be fused into a single CUDA kernel.
• Benefit: Minimizes latency and maximizes GPU utilization by keeping data in on-chip registers/caches.

TensorRT supports mixed-precision inference (FP32, FP16, INT8) to accelerate computation and reduce memory footprint. For INT8, it uses a calibration process to minimize accuracy loss.

• INT8 Quantization: Analyzes a representative dataset to determine optimal scaling factors (dynamic range) for each tensor.
• Benefit: Can provide up to 4x throughput increase and 4x memory reduction compared to FP32 with minimal accuracy degradation.

TensorRT selects the most efficient kernel implementation for each layer and target GPU architecture. It considers parameters like batch size, spatial dimensions, and precision to choose from a library of hand-optimized CUDA kernels.

• Architecture-Specific: Optimizes for the specific CUDA cores, Tensor Cores, and memory hierarchy of the deployment GPU (e.g., Ampere, Ada Lovelace).
• Benefit: Delivers peak hardware performance without manual kernel tuning by the developer.

TensorRT manages memory efficiently by reusing memory across tensors. It performs a memory planning step to allocate a single persistent workspace for intermediate activations.

• Reduced Allocations: Eliminates per-layer cudaMalloc calls during inference.
• Dynamic Shapes: Supports runtime dimension changes for inputs, allowing a single engine to handle variable batch sizes, image resolutions, or sequence lengths, which is crucial for real-time robotic perception.

The TensorRT runtime supports concurrent execution of multiple inference streams on a single GPU. This is essential for real-time systems that must process inputs from multiple sensors or pipelines in parallel.

• Overlap: Enables overlapping of data transfers (host-to-device, device-to-host) with kernel computation.
• Benefit: Maximizes throughput for batched requests and is foundational for sensor fusion pipelines in robotics.

A core model compression technique that reduces the numerical precision of a network's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This drastically reduces model size, memory bandwidth requirements, and computational cost. TensorRT performs post-training quantization and quantization-aware training to maintain accuracy. Key methods include:

• INT8 Quantization: Uses a calibration step to map float values to 8-bit integers with minimal loss.
• FP16 Precision: Utilizes half-precision floating-point for a balance of speed and accuracy on supported hardware.
• Sparsity: Exploits fine-grained structured sparsity in weights for further acceleration on Ampere GPUs and later.

The process of removing redundant or less important parameters from a neural network to reduce its computational footprint. TensorRT can leverage pruned models for faster inference. Common approaches include:

• Structured Pruning: Removes entire channels, filters, or neurons, leading to directly executable smaller networks.
• Unstructured Pruning: Sets individual weights to zero, creating sparsity that requires specialized hardware or libraries to exploit.
• Magnitude-based Pruning: Removes weights with the smallest absolute values. Pruning is often combined with fine-tuning to recover accuracy, and the resulting sparse models are ideal for TensorRT's optimization pipeline.

The foundational parallel computing platform and deep neural network library that TensorRT is built upon.

• CUDA (Compute Unified Device Architecture): NVIDIA's parallel computing platform and API model. TensorRT generates highly optimized CUDA kernels tailored for the specific network and target GPU.
• cuDNN (CUDA Deep Neural Network library): A GPU-accelerated library of primitives for deep neural networks. TensorRT often surpasses generic cuDNN performance by fusing layers and creating custom kernels that minimize memory transfers and maximize hardware utilization. Understanding these layers is key to advanced TensorRT profiling.