Glossary

MLPerf Inference

MLPerf Inference is an industry-standard benchmark suite that provides fair and reproducible performance comparisons of inference systems across a range of machine learning tasks, models, and deployment scenarios.

Get in touch Learn more

ML engineer working on model compression and quantization, laptop showing performance benchmarks, technical workspace.

INDUSTRY BENCHMARK

What is MLPerf Inference?

MLPerf Inference is the definitive industry-standard benchmark suite for evaluating the performance of machine learning systems when running trained models to make predictions.

MLPerf Inference is a rigorous, peer-reviewed benchmark suite that provides fair and reproducible comparisons of inference systems across diverse tasks, models, and deployment scenarios. Managed by the open engineering consortium MLCommons, it establishes standardized workloads—from computer vision and natural language processing to recommendation systems—to measure key metrics like latency and throughput under strict rules that ensure comparability. This allows engineers to objectively evaluate hardware accelerators, software frameworks, and full-stack solutions.

The benchmark defines multiple scenarios—such as single-stream, multi-stream, server, and offline—to reflect real-world use cases from edge devices to cloud data centers. Submissions are categorized into closed and open divisions, with the closed division requiring fixed model architectures and datasets to enable direct hardware/software comparisons, while the open division allows algorithmic optimizations. Results, which are audited for compliance, provide a critical, vendor-neutral reference for performance engineers and CTOs making infrastructure decisions based on efficiency and cost.

BENCHMARKING STANDARD

Core Characteristics of MLPerf Inference

MLPerf Inference is the definitive industry-standard benchmark suite for evaluating the performance, efficiency, and accuracy of systems running machine learning models in production. It provides rigorous, apples-to-apples comparisons across diverse hardware, software, and deployment scenarios.

Benchmark Scenarios

MLPerf Inference evaluates systems across three distinct deployment scenarios, each with strict rules to mirror real-world constraints:

Single-Stream: Measures latency for processing one sample at a time, critical for real-time applications like autonomous driving.
Multi-Stream: Simulates multiple cameras or sensors, requiring the system to process several independent data streams concurrently with latency bounds.
Server: Measures throughput under high query-per-second (QPS) load, representing cloud datacenter inference where batch processing and latency trade-offs are key.
Offline: Measures pure throughput for processing a fixed dataset as fast as possible, representing batch-oriented tasks like content moderation.

Model Suite & Workloads

The benchmark includes a curated suite of representative models spanning key AI domains to ensure comprehensive evaluation. Each workload tests different computational patterns:

Computer Vision: ResNet-50 (image classification), RetinaNet (object detection), 3D-UNet (medical image segmentation).
Natural Language Processing: BERT-Large (question answering, sentiment analysis).
Recommendation Systems: DLRM (deep learning recommendation model).
Speech Recognition: RNNT (recurrent neural network transducer).
Large Language Models: GPT-J (6B parameter model for text generation). Submissions must run the reference implementations without algorithmic changes, ensuring fairness.

Accuracy Targets & Validity

A core tenet is that performance measurements are meaningless without accuracy guarantees. Every submission must meet or exceed strict, predefined accuracy targets relative to the reference model's published baseline (e.g., 99% of the FP32 accuracy for ResNet-50 on ImageNet). The benchmark defines division categories:

Closed Division: Requires using the official model and dataset with fixed hyperparameters; only optimizations below the framework level (kernels, compilers) are allowed. This enables direct hardware comparison.
Open Division: Allows model modifications, pruning, and quantization, fostering innovation in model efficiency while still requiring accuracy compliance.

Measurement Methodology & Rules

Results are governed by a detailed v1.1 ruleset to ensure reproducibility and prevent gaming. Key methodological pillars include:

LoadGen: The standardized Load Generator is provided to all participants to issue queries according to the scenario specification, ensuring consistent load patterns.
Timing Boundaries: Measurements only count the "timing window" where the system is under steady load, excluding initialization and shutdown phases.
Statistical Validity: Runs must meet confidence interval requirements, often requiring multiple benchmark runs.
Auditability: Submissions include detailed scripts, documentation, and must be reproducible by the MLPerf organization.

System Categories & Tracks

Benchmarking is partitioned into tracks that reflect different system configurations and purposes, allowing relevant comparisons:

Chip-to-Cloud: From single accelerators (NVIDIA A100, Intel Habana Gaudi) to full servers and datacenter racks.
Edge vs. Datacenter: Separate categories for power-constrained edge devices (like Jetson AGX Orin) and high-power datacenter systems.
Network & Storage: Includes a network track for measuring inference across distributed systems and a storage track for data loading performance.
Mobile: Dedicated benchmarks for smartphone SoCs (Qualcomm Snapdragon, Google Tensor).

Performance-Per-Watt (Efficiency)

Beyond raw speed, MLPerf Inference mandates power measurement for most categories, establishing performance-per-watt as a critical efficiency metric. This directly addresses operational cost and sustainability.

Power Measurement: Requires using approved tools (e.g., NVIDIA NVML, Intel RAPL) to measure average power during the timing window.
Efficiency Scores: Results are often presented as samples per joule or inferences per kilowatt-hour, enabling CTOs to evaluate total cost of ownership.
Cooling Considerations: The rules account for system-level power, including cooling overheads in datacenter submissions.

INDUSTRY STANDARD

How MLPerf Inference Benchmarking Works

MLPerf Inference is the definitive industry-standard benchmark suite for evaluating the performance of machine learning systems during the prediction phase.

MLPerf Inference provides fair and reproducible performance comparisons by executing standardized benchmark tasks—such as image classification and natural language processing—across diverse deployment scenarios including data center, edge, and mobile. It measures key metrics like latency and throughput under strict rules that ensure comparability between different hardware and software stacks, preventing vendor-specific optimizations from skewing results.

The benchmark suite defines specific inference models, datasets, and quality targets that systems must meet, enforcing accuracy thresholds to prevent performance gains from sacrificing model correctness. Results are categorized into closed and open divisions, with the closed division requiring the use of reference models to highlight hardware and software efficiency, while the open division allows model modifications to foster innovation in model optimization techniques.

SCENARIO COMPARISON

MLPerf Inference Benchmark Scenarios

This table compares the four primary testing scenarios defined by the MLPerf Inference benchmark, which dictate the constraints on request arrival, batching, and system state during measurement.

Scenario	Request Arrival Pattern	System State	Primary Metric	Target Use Case
Single Stream	Single request processed in isolation	Cold start or warm	Latency (P90, P99)	Real-time, latency-critical applications (e.g., AR/VR, robotics)
Multi Stream	Multiple parallel, independent requests	Steady-state	Samples per second at target latency	Multi-camera, multi-sensor edge systems
Server	Requests arrive as a Poisson process	Steady-state	Queries per second (QPS) at target latency	Cloud-based model serving (e.g., REST API endpoints)
Offline	Entire dataset available at once	Cold start or warm	Samples per second	Batch processing, data analytics, maximum throughput analysis

MLPERF INFERENCE

Frequently Asked Questions

MLPerf Inference is the industry-standard benchmark suite for evaluating the performance and efficiency of systems running machine learning models in production. This FAQ addresses common technical questions about its methodology, components, and practical application.

MLPerf Inference is an industry-standard, peer-reviewed benchmark suite that provides fair and reproducible performance comparisons of systems running machine learning models. It works by executing a standardized set of benchmark tasks—such as image classification, object detection, and natural language processing—using prescribed reference models and datasets under strict rules that define the accuracy targets, load scenarios, and reporting requirements. Systems are measured on key metrics like throughput (samples per second) and latency (milliseconds) while meeting a minimum accuracy threshold, ensuring comparisons reflect real-world, production-ready performance.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

MLPERF INFERENCE ECOSYSTEM

Related Terms

MLPerf Inference is the cornerstone of objective AI system evaluation. These related concepts define the benchmarks, metrics, and deployment scenarios that constitute its standardized testing framework.

Benchmark Suite

The standardized collection of machine learning models, datasets, and accuracy targets that define an MLPerf Inference benchmark. Each suite targets specific domains:

Computer Vision: ResNet-50, RetinaNet, 3D-UNet.
Natural Language Processing: BERT-Large, GPT-J.
Recommendation Systems: DLRM.
Speech Recognition: RNNT. The suite provides the reference implementations, preprocessing rules, and quality thresholds that ensure fair, apples-to-apples comparisons across vastly different hardware and software stacks.

Submission Category

The classification of a benchmark result based on the constraints and optimizations allowed. MLPerf defines strict categories to ensure comparability:

Closed Division: The most restrictive. Uses only the reference model, disallowing changes to operators or numerical precision that affect accuracy. Measures pure hardware/software stack efficiency.
Open Division: Allows model modifications, pruning, and quantization, provided the submission meets the minimum accuracy target. Encourages algorithmic innovation and co-design.
Network Division: Benchmarks the entire system, including data center networking, for multi-node inference scenarios.
Edge Division: Sub-categories (Edge, Edge Offline, Edge Server) for on-device and localized inference.

Scenario

The operational context that defines how inference requests are presented to the system under test. Each scenario models a real-world deployment pattern:

Single Stream: Processes one request at a time. Measures latency for real-time applications (e.g., autonomous vehicle perception).
Multi Stream: Simulates multiple synchronous sensors (e.g., cameras). Measures the ability to meet latency deadlines for all concurrent streams.
Server: Models a cloud service with requests arriving asynchronously at a high rate. The primary metric is throughput at a target latency constraint (e.g., 99% of queries < 15ms).
Offline: Processes a fixed, batched dataset as fast as possible. Measures peak throughput for batch processing workloads.

Load Generator

The software component that emulates client request traffic according to a specific scenario. It is a critical part of the benchmark harness, responsible for:

Issuing inference queries with precise timing.
Measuring latency from issue to completion.
Adapting the request rate in Server scenario to find the maximum throughput under the latency constraint.
Ensuring the test meets minimum duration and sample count requirements for statistical validity. The reference implementation is provided by MLPerf to eliminate variance from custom load testing tools.

System Under Test (SUT)

The complete hardware and software stack being evaluated. This encompasses everything from the silicon to the application layer:

Hardware: CPUs, GPUs, TPUs, NPUs, memory, and storage.
System Software: OS, drivers, kernel.
Machine Learning Frameworks: TensorFlow, PyTorch, ONNX Runtime.
Compilers & Runtimes: TensorRT, OpenVINO, XLA, TVM.
Optimization Libraries: cuDNN, oneDNN. The SUT configuration must be fully documented in a submission, including software versions and critical system settings, to ensure reproducibility.

Inference Harness

The standardized software wrapper that integrates the Load Generator, Benchmark Suite, and System Under Test. It provides the controlled execution environment that guarantees a fair test:

Manages the lifecycle of the benchmark: setup, warm-up, timed test, and tear-down.
Handles dataset loading and preprocessing according to spec.
Validates output accuracy against the reference.
Collects and reports all required performance metrics (latency, throughput) and power measurements.
Ensures compliance with all benchmark rules. Submitters must use the official harness, preventing custom instrumentation from skewing results.

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.