AI Factory

Unlike traditional CPU-centric data centers operating at 5-10 kW per rack, an AI Factory is engineered for GPU clusters that routinely demand 40-100+ kW per rack. This requires a fundamental redesign of power distribution from the utility feed to the chip.

• Utility High-Voltage Ingress: Direct 138kV+ transmission-level connections to minimize transformation losses.
• Dense Busway Systems: Overhead power distribution replacing traditional under-floor cabling to handle high amperage.
• Per-Rack Power Shelves: Localized rectification and voltage regulation to step down power efficiently at the point of load.

Air cooling is thermodynamically insufficient for the TDP (Thermal Design Power) of modern accelerators. AI Factories mandate Direct Liquid Cooling to remove heat at the source, enabling higher rack densities and reducing facility overhead.

• Direct-to-Chip Cold Plates: Micro-channel copper plates affixed directly to GPUs and CPUs, circulating a dielectric coolant.
• Immersion Cooling: Submerging entire server sleds in a thermally conductive, non-conductive fluid for uniform heat extraction.
• CDU (Coolant Distribution Units): In-row or in-rack systems that manage the secondary fluid loop, isolating the facility water from the sensitive compute loop.

The AI Factory network is a back-end scale-out fabric distinct from the front-end internet connection. It must provide lossless, high-bandwidth connectivity for RDMA (Remote Direct Memory Access) traffic between thousands of accelerators.

• Spine-Leaf Topology: A Clos-based architecture where every leaf switch connects to every spine, ensuring predictable east-west bandwidth.
• InfiniBand or RoCEv2: The physical and link-layer protocols providing the low-latency, lossless transport required for NCCL collective operations.
• DCQCN (Data Center Quantized Congestion Notification): A critical congestion control algorithm to prevent packet loss in RDMA fabrics, which would otherwise cause catastrophic training stalls.

Training a foundation model requires a storage architecture that can saturate the ingest bandwidth of thousands of GPUs simultaneously. This is achieved through a parallel distributed file system with a direct path to GPU memory.

• Lustre or GPFS: High-performance parallel file systems that stripe data across hundreds of storage targets to provide aggregate throughput in the terabytes-per-second range.
• GPU Direct Storage (GDS): A technology enabling a direct DMA data path from NVMe drives or storage arrays into GPU memory, completely bypassing the CPU and system memory buffer.
• Checkpointing Performance: The storage system must sustain massive write bursts to snapshot the entire model state (potentially terabytes) in under a minute to minimize training downtime.

To eliminate the overhead of a hypervisor and ensure non-uniform memory access (NUMA) alignment, AI Factories provision physical servers directly via bare-metal orchestration platforms.

• Redfish API: An open standard for out-of-band management used to automate firmware updates, BIOS configuration, and power cycling of GPU nodes.
• GPU Operator: A Kubernetes operator that automates the lifecycle of NVIDIA drivers, the CUDA toolkit, and monitoring agents, ensuring a consistent software state across the entire fleet.
• Declarative Provisioning: Defining the entire node state—from firmware version to OS image—as code, ensuring every node is an immutable, reproducible unit of compute.

At the scale of thousands of GPUs, component failure is a statistical certainty, not a possibility. An AI Factory integrates comprehensive RAS (Reliability, Availability, Serviceability) features and real-time telemetry to manage hardware health.

• DCGM (Data Center GPU Manager): A suite of tools for active health monitoring, diagnostics, and performance analysis of NVIDIA GPUs, integrated into cluster-wide dashboards.
• GPU Burn-in Testing: A rigorous pre-production stress test that runs every new node at maximum thermal and computational load for an extended period to force early-life failures before the system enters the production pool.
• Predictive Maintenance: Analyzing telemetry streams to forecast component degradation (e.g., memory ECC error rate increases) and proactively schedule replacements before a hard failure causes a training job interruption.