Global Memory (Device Memory)

Global memory serves as the main data workspace for GPU kernels. It holds the input data, intermediate results, and final outputs for parallel computations. Unlike shared memory, which is private to a thread block, global memory is accessible by all threads across all blocks launched on a GPU. All data transferred from the host (CPU) via cudaMemcpy resides here, and kernels read from and write to this memory to perform their computations.

Global memory provides extremely high bandwidth (hundreds of GB/s to over 1 TB/s on modern GPUs) but also has high latency (hundreds of clock cycles). This latency is mitigated through massive parallelism: while one warp of threads is stalled waiting for data, the GPU scheduler can switch to other warps that are ready to execute, effectively hiding the latency. Achieving peak bandwidth requires coalesced memory access patterns, where consecutive threads in a warp access consecutive memory addresses.

Modern GPUs manage global memory through a virtual memory system, similar to CPUs. This enables key features:

• Unified Virtual Memory (UVM): Creates a single virtual address space shared by CPU and GPU.
• Demand Paging: Data is migrated to GPU memory only upon access (page fault).
• Memory Overcommit: Allows allocation of more memory than physically exists on the GPU, with pages swapped to host memory or storage. This abstraction simplifies programming but introduces overhead for page fault handling and migration.

Performance is heavily dependent on memory access patterns. Optimal patterns maximize throughput:

• Coalesced Access: Consecutive threads access consecutive 32-byte aligned segments. This merges requests into a minimal number of transactions.
• Strided or Misaligned Access: Threads access non-consecutive addresses, causing multiple, smaller transactions and drastically reducing effective bandwidth.
• Partition Camping: Excessive accesses to a single memory channel can create a bottleneck, even if accesses are coalesced. Profiling tools like Nsight Compute are essential for identifying suboptimal patterns.

Global memory sits at the bottom of the GPU's memory hierarchy, above system memory and storage but below on-chip memories. It is cached by several structures:

• L2 Cache: A large, unified cache shared by all Streaming Multiprocessors (SMs) that helps capture spatial and temporal locality.
• Read-Only Cache: A dedicated cache for data marked as constant (__constant__) or accessed via specific intrinsics.
• Texture Cache: Optimized for 2D spatial locality. Data not found in these caches must be fetched from global memory at full latency. Effective use of shared memory as a programmer-managed cache is the primary strategy to reduce global memory traffic.

Dynamic allocation in global memory (e.g., cudaMalloc) is expensive and can lead to memory fragmentation. Over time, free memory can become divided into small, non-contiguous blocks, preventing the allocation of large buffers even if total free memory is sufficient. Best practices to mitigate this include:

• Using memory pools (arenas) for frequent allocations/deallocations.
• Pre-allocating large buffers and managing sub-allocations manually.
• Leveraging stream-ordered memory allocators (like cudaMallocAsync) for asynchronous, context-aware allocation that reduces fragmentation. Fragmentation can lead to out-of-memory errors and reduced performance in long-running inference services.

A memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU. This allows both processors to access the same memory pointers using a common page table, simplifying data sharing and enabling on-demand page migration.

• Key Benefit: Eliminates the need for explicit cudaMemcpy calls for many data transfers.
• Mechanism: Managed by the system's Memory Management Unit (MMU), which handles GPU page faults by migrating pages between host and device memory.

A 3D-stacked memory technology where DRAM dies are vertically integrated with a GPU processor using Through-Silicon Vias (TSVs). It is the physical hardware most commonly used for global memory in modern high-performance accelerators.

• Architecture: Stacks memory directly on the processor interposer, providing extremely wide memory buses (1024-bit to 4096-bit).
• Advantage: Offers significantly higher bandwidth (over 1 TB/s) and better energy efficiency compared to traditional GDDR memory.
• Use Case: Essential for feeding data-hungry workloads in AI training and large-model inference.

A small, software-managed, on-chip cache memory on a GPU that is shared between all threads within a thread block. It acts as a programmer-controlled scratchpad to reduce accesses to the higher-latency global memory.

• Performance: Offers latency comparable to L1 cache and bandwidth an order of magnitude higher than global memory.
• Key Optimization: Avoiding bank conflicts, where multiple threads access the same memory bank, which serializes access.
• Typical Use: Staging data for cooperative algorithms like matrix multiplication or parallel reductions.

Host (CPU) memory that is prevented from being paged out to disk by the operating system. This enables high-bandwidth Direct Memory Access (DMA) transfers between the host and global memory.

• Function: Allows the GPU's DMA engine to access host memory directly without CPU intervention.
• Impact: Can double the bandwidth of host-to-device memory copies compared to using standard, pageable host memory.
• Trade-off: Overuse can reduce available system memory for other processes and harm overall system performance.

The organization of memory subsystems in a GPU into multiple levels with differing capacities, latencies, and bandwidths. Efficient programming requires placing data in the optimal level of this hierarchy.

Levels (from fastest/smallest to slowest/largest):

• Registers: Private to each thread, fastest access.
• Shared Memory / L1 Cache: Shared per thread block.
• L2 Cache: Shared across all Streaming Multiprocessors (SMs) on the GPU.
• Global Memory (Device Memory): Large, high-bandwidth, high-latency DRAM.
• System Memory & Storage: Accessed via Unified Virtual Memory or GPU Direct Storage.

An optimal pattern for reading from or writing to global memory where consecutive threads in a warp (a group of 32 threads) access consecutive memory addresses. This allows the memory controller to combine the accesses into a single, wide transaction.

• Mechanism: If thread T accesses address A, thread T+1 should access A+1 (for a 32-bit word).
• Benefit: Maximizes effective memory bandwidth, often achieving near-peak theoretical performance.
• Non-Coalesced Access: Scattered, unaligned accesses by a warp can result in 32 separate transactions, crippling performance.