NVIDIA Triton Inference Server

Triton supports a wide range of AI frameworks simultaneously, eliminating the need for separate serving infrastructure for different model types.

• Supported frameworks: TensorFlow, PyTorch, ONNX Runtime, TensorRT, OpenVINO, Python, and custom C++ backends
• Concurrent execution: Run a PyTorch vision model and a TensorFlow NLP model on the same GPU server
• Backend API: Extend Triton with custom logic using the backend API for proprietary runtimes
• Version management: Load multiple versions of the same model simultaneously for A/B testing and staged rollouts

Triton automatically groups individual inference requests into optimal batch sizes to maximize GPU utilization without requiring client-side batching logic.

• Adaptive batching: Dynamically adjusts batch size based on incoming request rate and latency targets
• No client changes: Clients send individual requests; Triton handles aggregation transparently
• Configurable delay: Set a maximum batching delay to bound latency while still achieving high throughput
• Shape-aware: Handles variable-shaped inputs within batches for models that support dynamic shapes

Chain multiple models into a directed acyclic graph (DAG) where the output of one model feeds directly into the input of the next, all orchestrated server-side.

• Pipeline definition: Define ensembles declaratively in a configuration file without writing glue code
• Reduced latency: Intermediate tensors stay on the GPU, eliminating costly host-device transfers between stages
• Example pipeline: Image preprocessing → object detection → classification → post-processing, all within one ensemble
• Parallel branches: Execute independent model paths concurrently within the same ensemble for complex workflows

Triton can execute multiple models simultaneously on the same GPU, sharing compute resources intelligently rather than dedicating hardware to a single model.

• GPU sharing: Run a lightweight classification model alongside a large language model on one accelerator
• Model priority: Assign relative priorities to models so critical inference paths receive preferential scheduling
• Rate limiting: Configure per-model rate limits to prevent resource starvation in multi-tenant deployments
• Instance groups: Specify how many parallel instances of a model run on each GPU or CPU for fine-grained resource control

A profiling tool that systematically searches the configuration space to find the optimal deployment parameters for each model on your specific hardware.

• Automated tuning: Sweeps across batch sizes, instance counts, and precision levels to maximize throughput under latency constraints
• Hardware-specific: Generates configurations tailored to your exact GPU model, not generic defaults
• Performance reports: Produces detailed latency-vs-throughput curves to inform capacity planning decisions
• Constraint-based: Define a maximum latency budget, and Model Analyzer finds the highest-throughput configuration that satisfies it

Triton can cache inference responses for identical or semantically similar inputs, dramatically reducing compute costs for repetitive queries.

• Exact match caching: Returns cached results when input tensors are byte-for-byte identical to a previous request
• Configurable TTL: Set time-to-live for cached entries to balance freshness against cache hit rate
• Shared cache: Multiple model instances share a single cache, maximizing reuse across the deployment
• Use case: High-traffic recommendation systems where many users receive identical model outputs for common inputs