How to Architect for Modular AI Hardware Components

The Open Compute Project provides open-source hardware designs and specifications that are the de facto standard for modular data center infrastructure. Adopting OCP designs is the single most effective step toward hardware longevity.

• Open Rack v3 (ORv3) defines a standardized rack and power distribution system.
• Open System Firmware (OSF) ensures firmware compatibility across vendors.
• Modular GPU designs like the HGX baseboard allow for accelerator upgrades without replacing the entire server chassis. Vendors like Wiwynn, Inspur, and Meta build production-ready OCP servers.

Open19 is a complementary standard focused on creating a common, tool-less 'sled' form factor for servers, storage, and networking within a 19-inch rack. It enables hot-swappable components at the server level.

• The brick cage and power distribution bus bar eliminate individual power supplies and cables per server.
• Components slide in and connect via a blind-mate connector, drastically reducing service time.
• This design is ideal for creating disaggregated pools of compute, memory, and storage that can be independently scaled. Vendors include LinkedIn (founder), HPE, and Wortmann AG.

CXL is an open industry interconnect standard that enables memory pooling and sharing across CPUs, GPUs, and other accelerators. This is critical for modular architectures.

• CXL 3.0 supports memory pooling, allowing a server to access a shared pool of DDR or persistent memory.
• This enables independent scaling of compute and memory, letting you upgrade CPUs without being forced to replace expensive, high-capacity DIMMs.
• Early adopters include Intel (with CXL-enabled CPUs), AMD, and memory vendors like Samsung and Micron.

NVMe-of decouples high-performance storage from individual servers, creating a shared, network-accessible storage pool. This is the storage counterpart to CXL for memory.

• RoCE (RDMA over Converged Ethernet) is the most common transport, providing low-latency remote direct memory access.
• Allows you to upgrade or replace compute nodes without migrating local storage, preserving data and reducing downtime.
• Key tools include the Linux NVMe-of target and initiator, and commercial solutions from Pure Storage, Dell, and IBM.

Redfish is a RESTful API standard (DMTF) for managing modern data center hardware. It provides a unified interface for monitoring and controlling modular components.

• Use it to query component health (fans, power supplies, temperatures), perform firmware updates, and control power states.
• Essential for automating the discovery and integration of hot-swapped components in a modular chassis.
• Supported by nearly all major server vendors, including HPE iLO, Dell iDRAC, and Supermicro.