ONNX Runtime

ONNX Runtime provides a unified interface to execute models across diverse hardware backends via Execution Providers (EPs). This abstraction allows the same ONNX model to run optimally on:

• NVIDIA CUDA and TensorRT for GPU acceleration.
• Intel OpenVINO for Intel CPUs and integrated graphics.
• DirectML for Windows with DirectX 12-capable hardware.
• CPU (default) with optimizations for x64 and ARM architectures.
• CoreML for Apple Silicon and iOS devices. The runtime automatically selects and manages the optimal execution path, enabling write-once, deploy-anywhere model serving.

A core feature for the Mixed Precision Inference pillar, ONNX Runtime provides extensive tools to reduce model precision for faster execution and lower memory use.

• Static Quantization: Converts FP32 models to INT8 using a calibration dataset. ONNX Runtime supports both symmetric and asymmetric quantization schemes.
• Dynamic Quantization: Quantizes activations on-the-fly at runtime, ideal for models like LSTMs and transformers where activation ranges vary.
• Quantization-Aware Training (QAT): Supports importing models trained with fake quantization nodes.
• FP16/BF16 Execution: Leverages Automatic Mixed Precision (AMP) on supported hardware (e.g., NVIDIA Tensor Cores) to speed up operations while maintaining accuracy. The onnxruntime.quantization module provides APIs for these processes.

ONNX Runtime is designed for integration into diverse production environments.

• Multi-Language APIs: First-class support for Python, C++, C#, Java, and JavaScript.
• Custom Operator Library: Developers can implement and register custom operators (kernels) for unsupported or proprietary operations, extending runtime functionality.
• Server Integration: Easily embedded into high-scale serving frameworks like Triton Inference Server or MLflow.
• Mobile Deployment: Lightweight builds (ONNX Runtime Mobile) for Android and iOS enable efficient on-device inference. This extensibility makes it a versatile backbone for both cloud and edge AI deployment pipelines.

The Open Neural Network Exchange (ONNX) is an open standard format for representing machine learning models. It enables interoperability between different frameworks like PyTorch, TensorFlow, and scikit-learn. ONNX Runtime executes models in this format.

• Portability: A model trained in one framework can be exported to ONNX and run in another environment.
• Graph Representation: Models are stored as computational graphs of operators, allowing for high-level optimizations.

TensorRT is NVIDIA's high-performance SDK for deep learning inference on NVIDIA GPUs. It performs graph optimizations, kernel auto-tuning, and precision calibration (INT8/FP16) to maximize throughput and minimize latency. While ONNX Runtime is cross-platform, it can use TensorRT as a hardware-specific execution provider to leverage these GPU optimizations for NVIDIA hardware.

Quantization is a model compression technique that reduces the numerical precision of weights and activations (e.g., from FP32 to INT8). ONNX Runtime provides extensive quantization tooling.

• Post-Training Quantization (PTQ): Converts a pre-trained model using a calibration dataset.
• Quantization-Aware Training (QAT): Trains a model with simulated quantization for higher accuracy.
• Execution Providers: ONNX Runtime applies quantized models efficiently via providers like CPU (MLAS) or GPU (TensorRT/CUDA).

ONNX Runtime is the inference engine within larger model serving architectures. These systems handle scaling, load balancing, and version management.

• Triton Inference Server: NVIDIA's serving platform that can use ONNX Runtime as a backend.
• KServe/Kubernetes: Cloud-native frameworks for deploying ONNX models as scalable microservices.
• Batching & Streaming: Serving layers implement continuous batching to group requests, which ONNX Runtime executes efficiently.