High Bandwidth Memory (HBM)

The defining feature of HBM is its 3D-stacked architecture. Multiple DRAM dies are vertically stacked and connected using Through-Silicon Vias (TSVs)—microscopic vertical conduits that pass through the silicon. This creates an extremely dense, short-path interconnect between memory layers.

• Key Benefit: Drastically reduces the physical distance signals must travel compared to planar GDDR layouts on a PCB.
• Result: Enables much higher data transfer rates at significantly lower power per bit.

HBM employs an exceptionally wide parallel bus—typically 1024 bits per stack—compared to the 32-bit wide interface of a GDDR6 memory chip. This width is the primary driver of its massive bandwidth.

• Mechanism: Bandwidth = Interface Width × Clock Speed. HBM trades high clock speeds (~1-2 GHz) for extreme width.
• Comparison: A single HBM3 stack with a 1024-bit bus at 1 GHz provides ~1 TB/s of bandwidth. Achieving this with GDDR6 would require prohibitively high, power-inefficient clock speeds.
• Implication: The wide interface requires advanced 2.5D packaging (like a silicon interposer) to route the thousands of connections to the GPU die.

HBM cannot be connected via a traditional motherboard PCB due to its ultra-wide bus. It requires 2.5D packaging using a silicon interposer.

• Interposer: A passive silicon slab that sits between the GPU compute die and the HBM stacks. It contains dense, high-speed wiring traces.
• Function: The GPU and HBM stacks are placed side-by-side on the interposer and connected via microbumps. The interposer routes the thousands of signals between them over very short distances (<1 mm).
• Benefit: Enables the necessary signal integrity for the wide, high-speed HBM interface. This co-packaging is a key differentiator from socketed GDDR memory.

To improve granularity and efficiency, each HBM physical channel is subdivided into two independent pseudo-channels.

• Structure: A 1024-bit HBM3 interface is typically organized as 16 independent channels, each 64 bits wide. Each of these is split into two 32-bit pseudo-channels.
• Advantage: Allows two memory controllers to operate on different banks within the same DRAM stack simultaneously, improving concurrency and reducing latency for smaller, non-sequential memory requests common in GPU workloads.
• Use Case: Essential for efficiently serving the highly parallel, random-access patterns of AI inference and training.

HBM is designed for superior energy efficiency per bit transferred, measured in picojoules per bit (pJ/bit).

• Primary Drivers:
  • Lower Voltage: HBM operates at lower voltages (~1.2V) compared to GDDR6 (~1.35V).
  • Short Interconnects: TSVs and the interposer minimize capacitive loading and signal power.
  • Moderate Clock Speeds: The wide interface reduces the need for power-hungry, ultra-high clock speeds.
• Impact: This efficiency is critical for data center GPUs where power consumption directly translates to operational cost and thermal design limits. HBM can deliver more bandwidth within a strict power envelope.

The HBM interface is managed by a sophisticated memory controller on the GPU die, designed to exploit HBM's internal bank group architecture.

• Bank Groups: HBM DRAM dies are organized into multiple bank groups that can be accessed independently. Operations to different bank groups can be pipelined.
• Controller Role: It schedules requests to maximize bank-level parallelism and minimize row activation (RAS) delays. It manages the pseudo-channels and handles command/address routing.
• Optimization: Effective use of bank groups hides the inherent latency of DRAM access, allowing the extremely high bandwidth of the physical interface to be realized in practice for real workloads.

HBM's defining feature is its 3D-stacked architecture. Multiple DRAM dies are vertically stacked on top of a base logic die containing the memory controller. They are interconnected using Through-Silicon Vias (TSVs)—microscopic vertical conduits that pass through the silicon layers. This stacking provides several key advantages:

• Extremely Wide Buses: HBM uses an interface that is 1024 bits wide per stack (compared to 32 or 64 bits for GDDR). Multiple stacks can be used in parallel.
• Reduced Physical Footprint: The vertical integration saves significant space on the processor package compared to planar GDDR memory chips arranged around the GPU.
• Shorter Interconnects: The proximity of the memory stack to the GPU logic die reduces signal path length, lowering power consumption and enabling higher speeds.

HBM is engineered for maximum bandwidth, which is the primary bottleneck for large neural network inference and training. Key performance differentiators include:

• Bandwidth: Current HBM3e implementations deliver over 1.2 TB/s per package. This is 3-5x higher than the best GDDR6X memory.
• Latency: While HBM provides exceptional bandwidth, its access latency (the time to fetch the first bit of data) is generally higher than GDDR due to its more complex, multi-banked architecture. However, the massive bandwidth often outweighs this for AI workloads characterized by large, sequential data transfers.
• Power Efficiency: HBM operates at a lower voltage (typically ~1.2V) and achieves higher bandwidth-per-watt than GDDR, a critical factor for dense server deployments.

HBM has evolved through several generations, each increasing density, speed, and efficiency:

• HBM1: The first generation, introduced with AMD's Fiji GPU. Offered up to 4 Hi (stack heights) and 128 GB/s per stack.
• HBM2: Became the standard for AI accelerators (e.g., NVIDIA V100, AMD MI50). Increased bandwidth to ~256 GB/s per stack and supported up to 8 Hi for higher capacity.
• HBM2e: An enhanced version with faster data rates, used in NVIDIA A100 and AMD MI100 GPUs.
• HBM3: Doubled the per-pin data rate and introduced features like on-die ECC. Used in NVIDIA H100 GPUs.
• HBM3e: The current leading-edge standard, pushing bandwidth beyond 1.2 TB/s per device and featured in NVIDIA's Blackwell B200 GPUs.

HBM is not just fast memory; it's an enabling technology for modern AI models. Its high bandwidth directly addresses the "memory wall" problem:

• Training: Efficiently feeds the colossal parameter sets (e.g., 175B+ parameters in GPT-3) and massive batch data into the GPU's compute cores during backpropagation.
• Inference: For large language models (LLMs), the KV Cache for the transformer's attention mechanism can consume 10s of GBs. HBM's bandwidth allows for rapid fetching of these cached keys and values, which is critical for low-latency token generation.
• Mixture of Experts (MoE): In sparse models like MoE, HBM bandwidth is crucial for rapidly loading the weights of the activated experts into the GPU's SRAM or caches.

HBM cannot be connected via traditional motherboard traces. It requires advanced 2.5D packaging.

• Silicon Interposer: A passive silicon layer placed between the GPU die and the HBM stacks. It contains ultra-dense wiring that connects the GPU's memory controller PHYs to the HBM stacks with minimal distance and loss.
• CoWoS (Chip-on-Wafer-on-Substrate): NVIDIA's and TSMC's primary packaging technology for HBM integration. The GPU and HBM stacks are first placed on a silicon interposer wafer (Chip-on-Wafer), which is then bonded to a package substrate (on-Substrate). This creates a single, integrated module.
• This tight integration makes HBM a fixed component of the GPU package, unlike socketed GDDR memory.

HBM exists within a broader memory hierarchy. Optimizing AI workloads requires understanding its interaction with other tiers:

• On-Chip SRAM/Cache: GPU L1/L2 cache. Data must move from HBM into these faster, smaller caches for computation. HBM bandwidth determines how quickly this can happen.
• Unified Virtual Memory (UVM): Allows the GPU to access CPU system memory (DDR) as a slower, larger backing store. HBM acts as the high-performance tier in this unified space.
• NVLink: While HBM feeds a single GPU, NVLink provides high-bandwidth connectivity between GPUs, allowing them to pool their HBM resources for giant model training.
• Memory Tiering: In advanced systems, hot data resides in HBM, warm data in DDR memory, and cold data may be paged to storage via GPU Direct Storage (GDS).

The organization of memory subsystems into multiple levels with differing performance characteristics. In a GPU system, this typically includes:

• Registers: Fastest, per-thread storage.
• Shared Memory / L1 Cache: Low-latency, software-managed cache shared by threads in a block.
• L2 Cache: Larger, unified cache for all streaming multiprocessors.
• HBM / GDDR (Global Memory): High-bandwidth, high-capacity DRAM (like HBM).
• Host Memory (DDR): System RAM, accessible via PCIe.
• Storage (NVMe): Slowest tier, used for oversubscription. Optimizing data movement through this hierarchy is fundamental to performance.

A digital circuit on the GPU that manages the flow of data to and from the DRAM (e.g., HBM stacks). It is a key determinant of achievable bandwidth and latency. Key functions include:

• Translating memory requests into DRAM commands (activate, read, write, precharge).
• Managing bank scheduling to maximize parallelism and minimize access conflicts.
• Implementing error correction code (ECC). The high bandwidth of HBM is enabled by a wide interface managed by many stacked memory controllers, each serving a segment of the memory.

An optimal pattern for threads in a GPU warp to access global memory (like HBM). Maximum bandwidth is achieved when consecutive threads access consecutive memory addresses in a single, aligned transaction.

• Coalesced: Threads 0-31 access addresses A, A+1, A+2,... A+31 → One 128-byte transaction.
• Uncoalesced: Threads access scattered addresses → Many smaller transactions, severely reducing effective bandwidth. Efficient kernels are designed to ensure memory access patterns are coalesced to fully utilize the available HBM bandwidth.

A system architecture that organizes heterogeneous memory types (HBM, DDR, CXL-attached memory, NVMe storage) into a performance/capacity hierarchy. Data is automatically promoted to faster tiers (e.g., HBM) or demoted to slower, larger tiers based on usage frequency ('hotness').

• Hot Data: Frequently accessed, resides in fast HBM.
• Cold Data: Infrequently accessed, migrates to host DDR or storage. This is enabled by technologies like Unified Virtual Memory (UVM) and is crucial for workloads whose working set exceeds physical HBM capacity.

The vertical electrical connection that passes completely through a silicon die or wafer. TSVs are the foundational technology enabling HBM by:

• Stacking DRAM dies vertically, creating a 3D structure.
• Providing a dense, short-path interconnect between the DRAM layers and the base logic die.
• Enabling the wide (1024-bit+), high-speed interface that defines HBM's bandwidth advantage over traditional 2D packaging with wire bonds. TSVs reduce power consumption and physical footprint compared to off-package interconnects.

A silicon layer, typically placed between the GPU/CPU package and the circuit board, that provides dense, high-speed wiring. In 2.5D packaging used for HBM:

• The GPU/CPU chip and HBM stacks are placed side-by-side on the silicon interposer.
• The interposer's microscopic wires connect the GPU's memory controllers to the HBM stacks with very short, high-bandwidth paths.
• This is distinct from 3D stacking (where dies are stacked directly on top of each other). The interposer is key to integrating HBM with the processor at acceptable yield and thermal limits.