Memory Profiler

A core function is tracking every memory allocation and deallocation call (e.g., cudaMalloc, cudaFree, malloc, free). By building a call graph, it identifies memory leaks—allocations that are never freed—and orphaned memory. This is critical for long-running inference servers where small, incremental leaks can lead to out-of-memory (OOM) errors over time. Profilers categorize leaks by size, allocation site, and lifetime.

This capability provides a real-time and historical view of the total memory footprint of an application or model. It breaks down usage by component:

• Model Weights: Static memory for parameters.
• Activations: Intermediate tensors during forward/backward pass.
• KV Cache: Memory for transformer attention keys and values.
• Workspace Memory: Temporary buffers for kernels (e.g., cuDNN, cuBLAS).
• Framework Overhead: Memory used by the deep learning framework itself. This breakdown is essential for understanding the peak memory requirement, which dictates the minimum GPU memory needed for a workload.

Beyond allocation size, profilers analyze how memory is accessed. This is vital for GPU optimization, where bandwidth is paramount. Key metrics include:

• Coalesced vs. Non-Coalesced Access: Identifies warps performing scattered reads/writes, wasting bandwidth.
• Shared Memory Bank Conflicts: Detects when multiple threads contend for the same memory bank, causing serialization.
• Cache Hit/Miss Rates: Measures effectiveness of L1/L2 cache usage.
• Memory Latency: Tracks the time spent waiting for data. Poor access patterns can be a primary source of inference latency, often more impactful than compute time.

Memory fragmentation occurs when free memory is broken into small, non-contiguous blocks, preventing allocation of larger tensors even if total free memory is sufficient. Profilers visualize the memory heap, showing allocated and free blocks over time. They help diagnose issues caused by:

• Frequent small allocations/deallocations.
• Out-of-order frees in stream-ordered allocators. Solutions often involve using memory pools (arenas) or adjusting allocator policies to reduce fragmentation overhead, which can otherwise degrade performance by up to 30%.

For systems using Unified Virtual Memory (UVM) or memory overcommit, profilers track page migration between GPU and host memory. They monitor:

• GPU Page Faults: Count and latency of faults triggered by accessing non-resident memory.
• Demand Paging: Traffic caused by data being paged in from system memory or storage.
• Thrashing: Excessive page migration that cripples performance. This analysis is critical for optimizing workloads that exceed physical GPU memory, helping to decide between buying more GPU RAM versus tuning UVM policies.

Advanced profilers correlate memory events with high-level model operations. They can attribute memory usage to specific neural network layers, operators, or training steps. For example, they can show that the memory spike during a forward pass is dominated by the attention layers' KV cache or a particular convolution's workspace. This layer-wise breakdown is indispensable for targeted optimization, such as applying activation checkpointing to specific layers or optimizing Mixture of Experts (MoE) routing memory.

A primary function is identifying memory leaks, where allocated memory is not properly released, leading to gradual exhaustion of available RAM or GPU memory. Profilers track allocations and deallocations, pinpointing the source code location where memory is allocated but never freed.

• Example: A CUDA kernel that allocates device memory with cudaMalloc but lacks a corresponding cudaFree in an error path.
• Tool Output: A profiler like NVIDIA Nsight Systems or Valgrind's Massif will show a monotonically increasing memory footprint over time, with a call stack pointing to the leaking allocation.

Profilers analyze memory access patterns within CUDA or ROCm kernels to identify suboptimal patterns that degrade bandwidth. They highlight issues like non-coalesced global memory accesses or shared memory bank conflicts.

• Key Metrics: Profilers report achieved memory bandwidth versus peak theoretical bandwidth and visualize access patterns.
• Actionable Insight: A profiler may show that threads within a warp are accessing scattered addresses, prompting a rewrite to ensure consecutive threads access consecutive memory addresses for coalesced access.
• Tools: NVIDIA Nsight Compute provides detailed per-kernel memory workload analysis.

Memory profilers help diagnose memory fragmentation, where free memory is broken into small, non-contiguous blocks, preventing large allocations despite sufficient total free memory. This is critical for long-running inference servers.

• Symptoms: Frequent out-of-memory errors or failed allocations for large tensors, even when the reported free memory seems adequate.
• Profiler Role: Visualizes the memory heap, showing allocated and free blocks. It can reveal "checkerboarding" where small free blocks are interspersed with allocated ones.
• Solution Guidance: Profiling data may lead to implementing a memory pool (arena allocator) to sub-allocate from large, pre-allocated blocks, reducing allocator overhead and fragmentation.

When using Unified Virtual Memory (UVM) or demand paging, profilers are indispensable for understanding page migration overhead. They track GPU page faults and data movement between CPU and GPU memory.

• Use Case: Identifying excessive page thrashing, where data is repeatedly migrated between memory tiers, causing severe latency.
• Profiler Data: Shows page fault counts, migration traffic (GB/s), and the specific memory addresses or kernels causing the faults.
• Optimization: Data may guide manual prefetching hints or adjustments to memory access patterns to improve locality and reduce faults.

Profilers measure the memory overhead introduced by deep learning frameworks (e.g., PyTorch, TensorFlow) and communication libraries (e.g., NCCL). This includes temporary buffers, gradient accumulators, and workspace memory.

• Example: Quantifying the extra memory consumed by PyTorch's Autograd engine for storing computation graphs versus inference-only mode (torch.inference_mode).
• Tool Example: PyTorch Profiler with memory tracing can attribute allocations to specific framework operations, helping choose more memory-efficient APIs or flags.
• Outcome: Enables informed trade-offs between functionality and memory footprint, crucial for fitting larger models onto a given GPU.

Before deployment, profilers provide a precise breakdown of a model's memory consumption during inference or training. This goes beyond simple parameter counting to include activations, optimizer states, KV caches for transformers, and framework overhead.

• Critical for LLMs: Profiling the KV cache memory usage per sequence is essential for optimizing continuous batching and determining maximum batch size.
• Output: A detailed table showing memory allocated for weights, forward activations, gradients, and optimizer states (e.g., Adam's m and v vectors).
• Decision Support: This data directly informs choices about model quantization, pruning, or the need for model parallelism.

A memory pool is a pre-allocated, contiguous block of memory from which smaller, dynamic allocations are sub-allocated. This technique is critical for GPU performance because:

• It drastically reduces the latency and overhead of frequent calls to system allocators like cudaMalloc and cudaFree.
• It mitigates memory fragmentation, a state where free memory is divided into small, non-contiguous blocks that prevent larger allocations.
• Pools are often stream-ordered, meaning allocations are associated with a specific CUDA stream for asynchronous, automatic reuse, minimizing synchronization stalls. A profiler helps identify when and where to implement pools by showing allocation frequency and lifetime patterns.

Memory fragmentation occurs when free memory is scattered into small, non-contiguous segments, preventing the allocation of larger blocks even if total free memory is sufficient. It is a primary target for memory profilers. There are two main types:

• External Fragmentation: Free memory is broken up between allocated blocks.
• Internal Fragmentation: Allocated memory is larger than requested, wasting space within the block. Fragmentation leads to out-of-memory errors, reduced performance, and inefficient use of GPU capacity. Profilers visualize fragmentation and track its cause, often pointing to suboptimal allocation strategies or patterns.

Page-locked memory (or pinned memory) is host (CPU) memory that is prevented from being paged out to disk by the operating system. This enables:

• High-bandwidth Direct Memory Access (DMA) transfers between host and GPU device memory.
• Asynchronous memory copies that overlap with kernel execution.
• Essential support for Unified Virtual Memory (UVM) and zero-copy transfers. While pinned memory accelerates data transfers, it is a scarce resource. A memory profiler can identify excessive or inefficient use of pinned memory, which can starve the host system and degrade overall performance.

Memory overcommit is a technique where the total memory allocated for active GPU workloads exceeds the physical GPU memory (VRAM) capacity. It relies on system RAM or storage (like NVMe SSDs) as a backing store, managed transparently by:

• Unified Virtual Memory (UVM) systems that provide a single address space.
• Demand paging and page migration, where data is moved on-demand upon a GPU page fault. While overcommit enables larger working sets, it can introduce severe latency penalties from swapping. A memory profiler is vital for monitoring access patterns, page fault rates, and thrashing behavior to determine if overcommit is beneficial or detrimental for a specific workload.

A stream-ordered memory allocator is a sophisticated GPU memory management scheme where allocations and deallocations are tied to the execution order of a specific CUDA stream. This enables:

• Automatic, asynchronous reuse: Memory freed by an operation in a stream can be immediately reused by a subsequent allocation in the same stream without global synchronization.
• Reduced fragmentation: Memory is reused in allocation order, promoting contiguous free blocks.
• Lower latency: Eliminates the need for explicit cudaFree calls and global synchronization points. Profilers like NVIDIA Nsight Systems can trace allocation lifetimes per stream, highlighting opportunities to adopt stream-ordered allocators for improved performance.

A GPU page fault is an exception triggered when a GPU kernel attempts to access a virtual memory address whose corresponding physical page is not resident in GPU memory. This occurs in systems with Unified Virtual Memory (UVM) or memory overcommit. The fault handler then:

1. Migrates the required page from host memory or storage (a process called page migration).
2. Resumes the kernel execution. While enabling flexible memory management, frequent page faults cause significant performance stalls. A memory profiler's most critical function is to quantify and locate these faults, identifying hot data that should be resident in GPU memory to avoid latency spikes.