GPU

The fundamental building block of an NVIDIA GPU. Each SM contains dozens of CUDA Cores for integer and floating-point arithmetic, Tensor Cores for matrix math, and dedicated Load/Store Units. An SM partitions threads into groups of 32 called warps, executing them in lockstep using a Single Instruction, Multiple Thread (SIMT) architecture. A single modern GPU can contain over 100 SMs, enabling the concurrent execution of thousands of threads.

Specialized execution units designed to accelerate matrix multiply-accumulate operations, the core computation of deep learning. Starting with the Volta architecture, Tensor Cores perform fused multiply-add on 4x4 matrices in a single clock cycle. Later generations added support for:

• FP16, BF16, TF32, FP8, and INT8 precision formats
• Sparsity acceleration, skipping zero-valued weights for a 2x throughput gain
• Structured sparsity patterns enforced during training fine-tuning

A multi-layered memory system designed to feed the compute engines with minimal latency:

• HBM3e: High-Bandwidth Memory stacks providing 4.8 TB/s+ bandwidth directly on-package
• L2 Cache: A large, shared on-die cache (tens of MB) reducing HBM traffic
• L1 Cache / Shared Memory: Per-SM programmable scratchpad memory for low-latency data sharing between threads in a block
• Register File: The fastest storage, with tens of thousands of 32-bit registers per SM

A high-bandwidth, low-latency GPU-to-GPU interconnect enabling direct memory access between accelerators. NVLink 4.0 provides 900 GB/s of bidirectional bandwidth per GPU. Multiple GPUs can be connected via NVSwitch to form a fully connected, non-blocking domain where every GPU can communicate with every other GPU at full speed. This is critical for tensor parallelism and pipeline parallelism strategies that split large models across multiple accelerators.

A hardware partitioning feature available on A100 and H100 GPUs that divides a single physical GPU into up to seven isolated instances. Each instance has:

• Dedicated portions of SMs, memory bandwidth, and L2 cache
• Hardware-enforced fault isolation and quality-of-service guarantees
• Independent execution of different workloads (inference, training, or multi-tenant serving)

This maximizes utilization in cloud and shared cluster environments without noisy-neighbor interference.

Dedicated hardware units that operate independently from the main compute pipeline:

• Copy Engines: Handle data transfers between host (CPU) and device (GPU) memory, and between GPUs, without stalling the SMs
• Compute Preemption: Allows long-running kernels to be paused at instruction-level granularity for higher-priority tasks

This enables overlapping computation with data movement, a critical optimization where the GPU computes one batch while the next batch is being transferred.

CUDA (Compute Unified Device Architecture) is NVIDIA's proprietary parallel computing platform and programming model. It exposes the GPU's massively parallel architecture to developers through extensions to standard languages like C++ and Python. Without CUDA, the GPU remains a graphics processor; with it, the device becomes a general-purpose vector supercomputer capable of executing thousands of threads simultaneously.

• Provides direct access to GPU virtual instruction set and memory
• Includes cuBLAS and cuDNN libraries optimized for deep learning
• Forms the moat that locks most AI training into the NVIDIA ecosystem

NVLink is a high-bandwidth, energy-efficient bidirectional interconnect that enables direct GPU-to-GPU communication without traversing the PCIe bus. NVSwitch extends this by connecting multiple GPUs in a non-blocking, full-bisection bandwidth fabric within a single node.

• NVLink 4.0 delivers 900 GB/s bidirectional bandwidth per GPU
• NVSwitch enables all-to-all communication across 8 GPUs simultaneously
• Eliminates the bottleneck of traditional PCIe switches for model parallelism

InfiniBand is the dominant high-performance interconnect for multi-node GPU clusters, providing ultra-low latency through RDMA (Remote Direct Memory Access). RDMA allows one GPU's memory to be written directly into another's across the network, bypassing the CPU and operating system entirely.

• Typical latency: sub-microsecond for small messages
• NCCL leverages InfiniBand for collective operations like all-reduce
• Essential for scaling training beyond a single 8-GPU node

Multi-Instance GPU is a hardware-level partitioning feature on NVIDIA A100 and H100 GPUs that carves a single physical GPU into up to seven isolated instances. Each instance receives dedicated compute cores, memory, and cache with hardware-enforced fault isolation.

• Enables secure multi-tenant GPU sharing without performance interference
• Each MIG instance appears as a fully independent GPU to CUDA
• Critical for inference serving where full GPU power is unnecessary

Direct Liquid Cooling (DLC) circulates a dielectric coolant directly to cold plates mounted on GPU dies, removing heat at the source rather than relying on ambient air. Modern H100 and B200 GPUs routinely exceed 700W TDP per module, making air cooling impractical at rack densities above 30kW.

• Enables 50kW+ per rack versus 15kW for air-cooled
• Reduces data center PUE below 1.1 by eliminating most fan power
• Required for high-density AI factory deployments

Slurm (Simple Linux Utility for Resource Management) is the dominant open-source workload manager for GPU clusters. It allocates exclusive or shared GPU resources to jobs, enforces NUMA alignment policies, and manages job queues with complex dependency graphs.

• Supports GPU Generic Resource (GRES) scheduling for precise allocation
• Integrates with enroot and Pyxis for container-native GPU jobs
• Powers many of the world's TOP500 supercomputers