Memory Controller

The controller receives read and write requests from the GPU's compute units and schedules them for execution on the DRAM. This involves critical decisions to minimize latency and maximize bandwidth.

• Arbitration Policies: The controller selects which pending request to service next. Common policies include First-Come, First-Served (FCFS), First-Ready, First-Come First-Served (FR-FCFS) which prioritizes requests to an already open DRAM row, and Oldest-Request-First to bound latency.
• Bank Management: DRAM is organized into independent banks. The controller manages bank activation, precharging, and row buffer states to hide the latency of opening and closing rows.
• Command Reordering: To maximize efficiency, the controller can reorder commands while respecting data dependencies, a process essential for achieving high utilization of the memory bus.

The controller translates the virtual or physical addresses from the GPU's request into the specific DRAM device coordinates: channel, rank, bank, row, and column. This mapping strategy profoundly impacts performance.

• Channel Interleaving: Addresses are striped across multiple memory channels to parallelize access and increase aggregate bandwidth.
• Bank/Row Mapping: How addresses map to banks and rows determines the potential for bank-level parallelism and the frequency of row buffer hits (fast) vs. row conflicts (slow). Poor mapping can lead to excessive row activations and high latency.
• Memory Partitioning: In unified memory systems (like NVIDIA's UVM), the controller works with the GPU's Memory Management Unit (MMU) to handle page faults and manage data placement across GPU memory, CPU memory, and even storage tiers.

DRAM cells are dynamic and leak charge; they must be periodically refreshed to retain data. The controller is responsible for generating all precise timing signals and interleaving refresh operations with normal accesses.

• Timing Parameters: The controller enforces a complex set of delays defined by the DRAM specification (e.g., tRCD - Row to Column Delay, tRP - Row Precharge time, tRAS - Row Active Time). Violating these timings corrupts data.
• Refresh Scheduling: The controller must issue a refresh command to each row every refresh interval (e.g., 64ms). It can use:
  • Burst Refresh: Stops all accesses for a refresh cycle, causing stalls.
  • Distributed Refresh: Interleaves refresh commands with normal traffic, which is standard in modern systems.
• Power Management: The controller can place DRAM into low-power states (e.g., self-refresh) during idle periods to save energy, adding wake-up latency when activity resumes.

To ensure data integrity against soft errors caused by cosmic rays or electrical noise, modern high-performance memory controllers implement Error-Correcting Code (ECC).

• On-Die ECC: Some GPU memory (like HBM2e) includes extra bits per channel for single-error correction, double-error detection (SECDED) managed directly by the memory die or the controller.
• Controller-Based ECC: The controller calculates and stores check bits alongside data in memory. When reading, it recalculates check bits and corrects single-bit errors or detects multi-bit errors.
• Impact on Bandwidth: ECC adds overhead—extra bits must be stored and transferred—which slightly reduces the effective user-accessible bandwidth compared to the raw physical bandwidth.

The DRAM controller does not operate in isolation; it is a key component in the broader memory hierarchy. Its efficiency is co-dependent with other subsystems.

• Cache Interaction: The controller's traffic is filtered by the GPU's L2 and L1 caches. High cache hit rates reduce demand on the controller, while cache misses become its requests. Cache eviction policies directly influence the stream of writebacks the controller must handle.
• Interconnect: The controller is connected to the GPU's streaming multiprocessors (SMs) via an on-chip network. The bandwidth and latency of this interconnect affect how quickly requests reach the controller.
• Unified Memory Management: In systems with Unified Virtual Memory (UVM), the controller works with the GPU MMU and page fault handler to manage data migration between GPU memory and page-locked host memory, making its role critical for zero-copy transfers and handling memory oversubscription.

High Bandwidth Memory is a 3D-stacked DRAM architecture where memory dies are vertically integrated with a GPU or CPU using through-silicon vias (TSVs). It provides the primary physical memory pool that a memory controller manages. Key characteristics include:

• Extremely high bandwidth (over 1 TB/s in modern systems) due to a wide interface.
• Improved energy efficiency compared to traditional GDDR memory.
• Stacked design reduces physical footprint on the processor package. The memory controller's design is tightly coupled to HBM's characteristics, managing its banks, rows, and channels to achieve peak performance for AI workloads.

Memory hierarchy organizes storage into multiple levels with different capacities, latencies, and bandwidths. A memory controller primarily manages the main DRAM tier (e.g., HBM), but its efficiency impacts the entire hierarchy. The typical GPU hierarchy includes:

• Registers: Fastest, thread-private storage.
• Shared Memory / L1 Cache: Low-latency, software-managed cache shared by threads in a block.
• L2 Cache: Larger, unified cache for all streaming multiprocessors.
• Global Memory (DRAM): High-capacity, high-bandwidth memory managed by the controller. The controller's ability to schedule requests efficiently helps hide the latency of accessing higher levels in this hierarchy.

Cache coherency is the property that ensures all processors and caches in a system have a consistent view of shared data. In systems with integrated GPU-CPU memory controllers (like AMD's APUs or some server CPUs), maintaining coherency is a critical controller function.

• Protocols: Implement protocols like MESI (Modified, Exclusive, Shared, Invalid) to track line states.
• Snooping: Controllers may 'snoop' on each other's memory traffic to invalidate or update cached copies.
• Impact on AI: For AI workloads sharing data between CPU and GPU, a coherent memory space managed by the controller simplifies programming but adds design complexity to the controller logic.

Non-Uniform Memory Access is a memory design where access time depends on the memory location relative to the processor. In multi-socket CPU systems or multi-GPU systems with complex interconnects, memory controllers manage these access asymmetries.

• Local vs. Remote Memory: A processor has faster access to memory attached to its local controller.
• Controller Role: The memory controller must correctly route requests to local or remote memory nodes.
• AI System Relevance: Large-scale AI training servers often have NUMA characteristics. Optimal data placement, guided by the OS and managed by the controllers, is crucial for performance.

A memory access pattern describes the order and structure in which threads read from or write to memory. The efficiency of a memory controller is heavily dependent on the pattern generated by the workload.

• Coalesced Access: The optimal pattern where consecutive threads access consecutive addresses, allowing the controller to service a warp's request in a minimal number of transactions.
• Strided or Random Access: Inefficient patterns that force the controller to issue many small, unaligned transactions, wasting bandwidth and increasing latency.
• Controller Scheduling: Advanced controllers use schedulers and reorder buffers to mitigate the impact of poor access patterns, but the kernel's inherent pattern is the primary determinant of performance.

Demand paging and page migration are virtual memory techniques that rely on the memory controller's ability to handle page faults. In unified memory systems (like NVIDIA's UVM), these are essential for oversubscribing GPU memory.

• Page Fault: Occurs when a GPU thread accesses a virtual address whose page is not in GPU physical memory. The memory management unit signals this to the controller/driver.
• Page Migration: The controller (via the driver) then orchestrates the transfer of the required page from host memory or storage into GPU memory.
• Controller's Role: While high-level management is software-driven, the hardware memory controller must efficiently integrate with this system, handling the physical movement of data triggered by faults.