InfiniBand

InfiniBand defines data rates in terms of Single Data Rate (SDR), Quad Data Rate (QDR), Fourteen Data Rate (FDR), Enhanced Data Rate (EDR), High Data Rate (HDR), and Next Data Rate (NDR). Each generation doubles the per-lane speed. Links are aggregated using 4x or 12x wide ports.

• Current Standard (HDR): Provides 200 Gb/s per port (using a 4x wide link at 50 Gb/s per lane).
• Aggregate Throughput: A 12x wide HDR link delivers 600 Gb/s of bidirectional bandwidth.
• Comparison: This significantly outpaces standard 100 Gb Ethernet, providing the raw throughput needed to prevent the network from becoming a bottleneck during all-to-all communication phases in large model training.

InfiniBand provides several transport services (Reliable Connected, Unreliable Datagram, etc.) tailored for different traffic patterns. It features sophisticated hardware-based congestion control that detects network hotspots and dynamically throttles traffic at the source, preventing performance collapse.

• Reliable Connected (RC): The most common service for HPC, providing guaranteed, in-order packet delivery with hardware-based acknowledgment.
• Adaptive Routing: Packets can take multiple paths through the fabric to balance load and avoid congested links.
• Priority-based Flow Control: Allows critical traffic (e.g., latency-sensitive synchronization messages) to preempt bulk data transfers.

InfiniBand networks can be logically subdivided into partitions (PKeys), creating isolated virtual fabrics on the same physical hardware. This allows a single cluster to securely host multiple tenants or job queues (e.g., different AI training jobs) without interference.

• Hardware Enforcement: Partition membership and traffic isolation are enforced at the network adapter level, providing strong security and performance guarantees.
• Quality of Service (QoS): Combined with Service Levels (SL), it allows administrators to assign bandwidth and priority guarantees to different partitions or traffic types, ensuring critical workloads meet service-level objectives.

InfiniBand is the de facto standard for connecting nodes in the world's fastest supercomputers. Its Remote Direct Memory Access (RDMA) capability is critical for Message Passing Interface (MPI) traffic, allowing scientific simulations—like climate modeling, astrophysics, and molecular dynamics—to scale across tens of thousands of servers with minimal communication overhead. The TOPS00 list of supercomputers is dominated by InfiniBand-based systems.

Training modern large language models (LLMs) and foundation models requires synchronizing terabytes of parameters across thousands of GPUs. InfiniBand's low latency and high bandwidth (up to 400 Gb/s per port) are essential for efficient all-reduce operations during distributed training. This minimizes the time GPUs spend waiting for gradient updates, directly reducing training time and cost. Major cloud providers offer InfiniBand-backed instances specifically for AI workloads.

In financial markets, latency is measured in microseconds. InfiniBand's sub-microsecond latency and deterministic performance are used to connect trading servers to exchange matching engines and for inter-server communication within risk analysis clusters. This enables algorithmic trading systems to execute orders and process market data faster than competing TCP/IP-based networks.

Running physics-based simulations for robotics, autonomous vehicles, or computational fluid dynamics requires massive parallel computation. InfiniBand connects the compute cluster running the simulation engine (e.g., NVIDIA Omniverse, Isaac Sim) to high-speed parallel file systems (like Lustre or GPFS) and between simulation worker nodes. This ensures that sensor data, state updates, and rendered frames can be exchanged in real-time, enabling hardware-in-the-loop (HIL) testing and digital twin synchronization.

Workloads like genomic sequence analysis, cryo-EM image processing, and molecular dynamics simulations generate and process petabytes of data. InfiniBand fabrics connect sequencers, imaging devices, and high-performance compute clusters to centralized storage, enabling rapid data movement for pipelines that identify genetic variants, model protein folding, or screen drug compounds. This accelerates research timelines in bioinformatics and pharmaceutical R&D.

A parallel file system is a type of storage architecture designed for HPC and AI workloads where hundreds or thousands of compute nodes need simultaneous, high-speed access to shared data. InfiniBand is the preferred network for connecting compute nodes to these storage systems. Examples include Lustre, IBM Spectrum Scale (GPFS), and BeeGFS.

These systems stripe data across multiple storage servers and metadata managers. InfiniBand's high throughput and RDMA capabilities are essential for:

• Saturating the bandwidth of many storage devices in parallel
• Providing low-latency metadata operations
• Enabling checkpoint/restart for long-running simulations by quickly writing application state to storage

A compute-bound workload is a task whose execution time is limited primarily by the speed of the processor (CPU/GPU), not by data transfer speeds (I/O-bound) or memory access (memory-bound). Training large AI models or running high-fidelity physics simulations are classic examples.

For such workloads to scale across thousands of nodes, the network must not become the bottleneck. InfiniBand is engineered to keep the workload compute-bound by providing:

• Bandwidth that matches or exceeds local PCIe speeds (e.g., NDR InfiniBand at 400 Gb/s)
• Latency low enough that nodes spend minimal time waiting for data from peers
• Efficient collective communication primitives (like All-Reduce) for distributed training