TensorRT

TensorRT parses a trained model and applies graph optimizations to restructure the compute graph for maximum efficiency. Key techniques include:

• Vertical Layer Fusion: Merges convolution, bias, and ReLU layers into a single kernel call, eliminating memory bottlenecks.
• Horizontal Layer Fusion: Combines layers with identical operations but different weights into one wider layer.
• Elimination of Concatenation Layers: Removes unnecessary transpose and concatenation operations by pre-allocating output buffers. These optimizations significantly reduce kernel launch overhead and memory bandwidth usage.

TensorRT supports reduced-precision inference to accelerate computation on modern GPUs.

• FP16 (Half-Precision): Leverages Tensor Cores on Volta and newer architectures for up to 2x speedup over FP32 with minimal accuracy loss.
• INT8 (Integer Quantization): Requires a calibration step using a representative dataset to determine optimal scaling factors for weights and activations. This can yield up to 4x throughput improvement.
• Explicit Quantization: Developers can manually define quantization ranges for fine-grained control over numerical precision.

TensorRT uses a custom memory allocator that minimizes the overhead of cudaMalloc and cudaFree calls during inference. The allocator:

• Pre-allocates a large memory pool at engine build time.
• Reuses intermediate tensors whose lifetimes do not overlap, reducing peak memory usage.
• Eliminates fragmentation by managing memory in a contiguous block. This is critical for real-time signal classification where deterministic latency is required and memory allocation jitter is unacceptable.

During the builder phase, TensorRT profiles the target GPU to select the fastest implementation for each layer. The auto-tuner:

• Benchmarks multiple kernel variants (e.g., Winograd, FFT, direct convolution) for the specific input dimensions and filter sizes.
• Selects the optimal tactic based on measured execution time, not heuristics.
• Caches these selections in a timing cache to accelerate subsequent builds for the same hardware. This ensures peak performance on the specific deployment GPU, whether it's a Jetson Orin or an A100.

TensorRT separates the build phase from the runtime phase. The optimized model is serialized into a plan file (also called an engine). This plan:

• Is binary, platform-specific, and tied to the exact GPU architecture it was built for.
• Can be deserialized and executed without re-optimization, enabling fast startup.
• Supports versioning to ensure compatibility checks between the serialized engine and the runtime library. This decoupling is essential for edge deployment where build-time optimization occurs on a development workstation and the plan is deployed to a fielded SDR system.

When TensorRT encounters an unsupported operation, it falls back to the original framework's runtime, which breaks the optimized graph. To avoid this, developers can implement custom plugins:

• Write CUDA kernels that implement the custom operation and register them with TensorRT.
• Define the plugin's input/output tensor shapes and data types via the IPluginV2 interface.
• Integrate custom signal processing blocks, such as a specialized FFT or cyclostationary feature extractor, directly into the optimized engine. This allows the entire modulation classification pipeline to run within a single, fused TensorRT context.