MLPerf

The Closed Division mandates the use of a mathematically equivalent model and optimizer to ensure direct hardware and software system comparisons. Submitters must use the reference implementation, such as a specific ResNet-50 or BERT variant, preventing accuracy-advantageous cheating. This division isolates the raw throughput and latency capabilities of the underlying infrastructure, making it the primary metric for hardware procurement decisions.

The Open Division allows unrestricted model architecture, learning rate schedules, and optimization techniques to push the absolute boundaries of speed. While not directly comparable between vendors, this division drives industry innovation by rewarding novel approaches like advanced pruning, neural architecture search, or low-precision arithmetic that achieve state-of-the-art training times or inference latency.

Training benchmarks measure the wall-clock time required to train a model to a target quality threshold. Key workloads include:

• Image Classification: ResNet-50 on ImageNet
• NLP: BERT on Wikipedia/BooksCorpus
• Recommendation: DLRM on Criteo Terabyte
• Reinforcement Learning: Mini-go
• Scientific: CosmoFlow These tests stress GPU interconnect bandwidth and all-reduce collective operations.

Inference benchmarks evaluate a system's ability to process prediction requests under strict latency constraints. The suite is split into two scenarios:

• Server Scenario: Maximizes throughput under a target latency budget (e.g., 99th percentile < 15ms), simulating cloud API backends.
• Offline Scenario: Maximizes throughput with no latency constraint, simulating batch processing. Workloads span vision, speech, and medical imaging.

The optional Power Division adds energy efficiency to the benchmark, measuring performance per watt. Systems are instrumented with precision power meters at the power supply unit or bus level. This metric is critical for large-scale AI factory deployments where operational expenditure is dominated by electricity costs and thermal management. It penalizes brute-force scaling in favor of performance-per-watt optimization.

MLPerf is governed by the MLCommons consortium, an open engineering group with members from academia and industry. Every submission undergoes a rigorous peer-review process where results are reproduced and validated by third parties. This prevents benchmark gaming and ensures that published numbers—whether for a single NVIDIA H100 or a 10,000-node cluster—are verifiable and trustworthy for enterprise procurement.