Memory Fragmentation

Fragmentation manifests in two primary forms. Internal fragmentation occurs when an allocated memory block is larger than the requested size, leaving unused memory within the allocated block. External fragmentation is the state where free memory is scattered as small, non-contiguous blocks between allocated blocks, preventing new large allocations despite adequate total free space. In GPU contexts, external fragmentation is the primary concern for large tensor allocations.

Fragmentation directly degrades allocator performance. Key overheads include:

• Search Latency: The allocator must spend more time searching through a fragmented free list to find a suitable block, increasing allocation time.
• Splitting & Coalescing: To service a request, a larger free block may be split, creating more small fragments. Conversely, freeing a block requires checking adjacent blocks to coalesce them into a larger free block, adding computational cost.
• Metadata Bloat: Each allocated and free block requires metadata (size, pointers). Proliferation of small blocks increases metadata overhead, consuming memory that could be used for data.

GPU kernels and libraries (e.g., cuBLAS, cuDNN) often require contiguous memory for large tensors, buffers, and model weights. Fragmentation is the primary cause of Out-of-Memory (OOM) errors even when the nvidia-smi tool shows significant free memory. This occurs because the largest contiguous free block (cudaMemGetInfo) is smaller than the requested allocation, causing allocation failure and crashing the workload.

Certain application patterns accelerate fragmentation:

• Variable-Lifetime Allocations: Mixing long-lived allocations (model weights) with many short-lived, variable-sized allocations (activations, temporary buffers).
• Non-Stream-Ordered Frees: Freeing memory out of the order it was allocated, which hinders the allocator's ability to coalesce adjacent free blocks.
• Peak Memory Usage Spikes: Workloads with highly variable memory demands create transient large allocations that, when freed, leave behind holes of varying sizes.

A primary defense against fragmentation is the use of a memory pool or arena. Instead of frequent cudaMalloc/cudaFree calls, the application requests a large block from the system upfront. Subsequent allocations are sub-allocated from this pool. Frees return memory to the pool, not the system. This confines fragmentation to the pool, prevents global OOM errors, and drastically reduces allocator overhead. Pools can be stream-ordered for asynchronous, lock-free management.

Fragmentation effects compound with other memory subsystems:

• Unified Memory (UVM): Fragmentation in the GPU's physical memory can increase page fault and page migration overhead as the driver struggles to manage non-contiguous page mappings.
• Peer-to-Peer (P2P) Access: Direct GPU-to-GPU transfers often require contiguous buffers. Fragmentation can prevent the creation of suitable buffers, forcing a fallback to a slower copy via host memory.
• Memory Overcommit: In oversubscribed systems, fragmentation reduces the efficiency of demand paging, as evicting and fetching small, scattered pages is less efficient than moving large, contiguous chunks.

A memory pool is a pre-allocated, contiguous block of memory from which smaller allocations are sub-allocated for application use. This strategy is a primary defense against fragmentation.

• Mechanism: Instead of frequent calls to system allocators (e.g., cudaMalloc/cudaFree), the application manages allocations from its own reserved pool.
• Benefit: Dramatically reduces allocator overhead and external fragmentation by reusing fixed-size blocks within the pool.
• Example: Frameworks like TensorRT and PyTorch's caching allocator use sophisticated pool-based strategies to manage GPU memory for deep learning tensors, grouping allocations by size to minimize wasted space.

Unified Virtual Memory is a memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU. It is foundational to modern memory oversubscription systems where fragmentation can have cascading effects.

• How it works: Both processors use the same virtual memory pointers. The physical backing for a page can reside in CPU RAM, GPU VRAM, or even storage.
• Fragmentation Context: UVM can mask device-level fragmentation by allowing allocations to be satisfied from host memory, but it introduces system-wide fragmentation concerns across the memory hierarchy. Poor allocation patterns can thrash the page migration system.

Memory overcommit is a technique where the total memory allocated to active GPU workloads exceeds the physical GPU memory capacity. It relies on Unified Virtual Memory and a backing store (system RAM, SSD) to enable larger working sets.

• Fragmentation Impact: Fragmentation in the GPU's physical memory is acutely problematic under overcommit. If free memory is fragmented, the system cannot efficiently page out large, contiguous idle regions, forcing it to migrate many small, active pages instead. This leads to excessive GPU page faults and severe performance degradation.
• Trade-off: Enables larger models but makes efficient, defragmented memory layout critical for performance.

A stream-ordered memory allocator is a GPU memory management scheme where allocations are tied to a specific CUDA stream and freed automatically when all work in that stream dependent on the allocation is complete.

• Fragmentation Mitigation: Allocations and frees happen in predictable order within a stream. This allows for bump allocation and immediate reuse of memory blocks without complex global tracking, significantly reducing external fragmentation.
• Use Case: NVIDIA's cudaMallocAsync and cudaFreeAsync implement this pattern. It is highly effective for inference servers using continuous batching, where request lifetimes are tied to computation streams.

Page migration is the process of moving memory pages between tiers of the memory hierarchy (e.g., GPU VRAM to CPU RAM). Demand paging is the specific technique of migrating pages only upon access attempt, triggered by a GPU page fault.

• Fragmentation Link: These mechanisms are how systems cope with the symptoms of fragmentation in a unified memory system. When a large contiguous allocation cannot be placed in GPU memory due to fragmentation, the system may use these techniques to assemble it virtually from non-contiguous physical pages across different memory tiers, at a cost of higher latency.
• Performance: Excessive migration due to fragmentation is a key performance limiter in oversubscribed systems.

The memory access pattern defines the order and structure in which threads read/write memory. While primarily a performance concept, it interacts with allocator behavior and fragmentation.

• Coalesced Access: The optimal pattern where consecutive threads access consecutive addresses. Allocators strive to provide contiguous memory blocks to enable this.
• Fragmentation Effect: If an allocator, due to fragmentation, cannot provide a contiguous buffer for a tensor, it may need to use a strided or non-contiguous layout. This forces uncoalesced access patterns, drastically reducing effective memory bandwidth and compounding the performance penalty beyond the allocator's own overhead.