ONNX Runtime

A specialized build of ONNX Runtime that compiles to WebAssembly and WebGPU, enabling client-side inference directly in the browser. This eliminates server round-trips and keeps user data local.

• WebAssembly Backend: CPU execution with optimized SIMD kernels
• WebGPU Backend: Leverages the browser's GPU API for accelerated inference
• WebGL Backend: Fallback for broader device compatibility
• Supports standard ONNX models with no conversion required

The pluggable hardware abstraction layer that allows ONNX Runtime to dynamically dispatch compute operations to different backends. Each EP encapsulates a specific hardware-specific library, enabling a single model to run optimally across diverse silicon without code changes.

• CPUExecutionProvider: Default x86/ARM optimization via MLAS
• CUDAExecutionProvider: NVIDIA GPU acceleration using cuDNN and TensorRT kernels
• TensorRTExecutionProvider: Maximizes throughput via NVIDIA's graph optimization compiler
• OpenVINOExecutionProvider: Targets Intel CPUs, GPUs, and VPUs
• DirectMLExecutionProvider: Hardware-agnostic acceleration on Windows via DirectX 12
• CoreMLExecutionProvider: Leverages Apple Neural Engine on macOS and iOS

ONNX Runtime applies a series of transformation passes to the computational graph before execution, eliminating redundant operations and fusing nodes for improved performance. These optimizations are configurable by level and are critical for reducing inference latency in resource-constrained air-gapped environments.

• Basic: Removes redundant casts, identity nodes, and dropout layers
• Extended: Fuses activation functions with preceding operations (e.g., Conv+ReLU)
• Layout Optimizations: Automatically converts tensors to hardware-preferred memory layouts (NHWC vs NCHW) for optimal cache locality
• All: Enables all available optimizations, including aggressive constant folding and node elimination

The Open Neural Network Exchange (ONNX) is an open standard for representing machine learning models as a directed acyclic graph of operators. It serves as the universal intermediate representation that decouples the training framework from the inference engine, enabling seamless portability across tools and hardware.

• Operator Set: A versioned collection of standard operations (e.g., Conv, MatMul, LSTM) with well-defined semantics
• IR Version: Tracks the evolution of the model representation itself, independent of operators
• External Data: Allows large weight tensors to be stored in separate files, keeping the model proto under 2GB for compatibility
• Custom Operators: Extends the standard set with domain-specific kernels, essential for deploying proprietary architectures in disconnected environments

ONNX Runtime provides built-in tools for reducing model precision from FP32 to lower-bit representations, dramatically decreasing memory footprint and inference latency. This is essential for deploying large language models on sovereign infrastructure with limited GPU memory.

• Dynamic Quantization: Quantizes weights to INT8 at load time while activations remain in floating point; zero calibration data required
• Static Quantization: Quantizes both weights and activations using a representative calibration dataset for maximum accuracy
• Quantization-Aware Training (QAT): Simulates quantization noise during training to produce models robust to INT8 or INT4 inference
• FP16 Conversion: Simple half-precision conversion for compatible hardware with minimal accuracy impact

ONNX Runtime serves as a first-class backend within NVIDIA Triton Inference Server, enabling production-grade model serving with dynamic batching, model ensembles, and GPU sharing. This integration is critical for building scalable inference pipelines in disconnected Kubernetes clusters.

• Concurrent Model Execution: Triton manages multiple ONNX Runtime instances across GPUs partitioned via NVIDIA MIG
• Dynamic Batching: Automatically coalesces individual inference requests into optimal batch sizes for maximum throughput
• Model Ensembles: Chains ONNX models with other framework backends (TensorRT, PyTorch) into a single inference DAG
• Business Logic Scripting: Python backend allows pre/post-processing logic to execute alongside ONNX Runtime models

Deploying ONNX Runtime in disconnected environments requires bundling the runtime binaries, execution provider libraries, and model artifacts into a self-contained package. This process eliminates any runtime dependency on external package repositories.

• Pre-built Binaries: Download platform-specific native packages (.whl, .deb, .rpm) from the official release page and transfer via data diode or physical media
• Execution Provider Dependencies: Manually bundle CUDA/cuDNN shared libraries for GPU execution providers, ensuring version compatibility with the target cluster's GPU Operator
• Containerized Runtime: Build a base Docker image with ONNX Runtime and all EPs, then push to a Private Container Registry within the air gap
• Model Serialization: Convert all models to ONNX format with external data files, then sign with a Software Bill of Materials (SBOM) for supply chain verification