Memory Cgroup (Control Group)

The core function of a memory cgroup is to enforce a hard limit on the total memory consumption of its member processes. This includes:

• RSS (Resident Set Size): Physical memory actively used.
• Page Cache: Memory used for cached files.
• Kernel data structures: Memory allocated for sockets, tmpfs, etc. When a cgroup exceeds its limit, the kernel invokes the OOM (Out-Of-Memory) killer to terminate processes within that cgroup, protecting the host system and other workloads from resource starvation.

Memory cgroups track and can limit the combined usage of physical memory and swap space. This is controlled by the memory.memsw.limit_in_bytes parameter.

• Key Distinction: A memory.limit_in_bytes controls only RAM. Adding a memory.memsw.limit_in_bytes prevents a cgroup from swapping excessively, which can cause severe I/O latency.
• Use Case: Critical for preventing a single container from causing system-wide thrashing by consuming all available swap, which would degrade performance for all other processes.

Memory cgroups are organized in a tree hierarchy, where child groups inherit and can further restrict the limits of their parent. This enables sophisticated resource delegation.

• Example: A root cgroup / has 64GB. A child /batch_jobs is limited to 48GB. A grandchild /batch_jobs/training is further limited to 32GB.
• Charge Propagation: Memory usage is charged to the cgroup where the page was allocated and to all its ancestors. This ensures hierarchical limits are enforced and usage is properly accounted up the tree.

A soft limit provides a hint to the kernel about desired memory usage, not a strict boundary. When system memory is under pressure, the kernel preferentially reclaims pages from cgroups that have exceeded their soft limit before targeting those below it.

• Behavior: This is a best-effort mechanism. A cgroup can exceed its soft limit if free memory is available.
• Purpose: Allows for overcommitment while establishing a priority for reclaim. Lower-priority batch jobs can be given a soft limit, allowing their cache to be reclaimed first when the system is busy, protecting latency-sensitive services.

Cgroups can generate pressure stall information (PSI) metrics, providing fine-grained visibility into resource contention.

• Measures: some (at least one task stalled), full (all non-idle tasks stalled).
• Metrics: memory.pressure file reports the percentage of time tasks in the cgroup were stalled waiting for memory in a given window (e.g., 10s, 60s, 300s).
• Application: Enables proactive monitoring and autoscaling. An orchestration system can observe rising some pressure and decide to schedule a pod on a different node before latency spikes occur.

Administrators can fine-tune the behavior of the OOM killer within a cgroup using the memory.oom_control interface.

• Disabling OOM Killer: Setting oom_kill_disable to 1 prevents processes in the cgroup from being killed. Instead, the kernel will block allocating processes, putting them into an uninterruptible sleep (D state) until memory is freed. Use with extreme caution as it can lead to a deadlock.
• OOM Score Adjustment: The oom_score_adj parameter for individual processes influences their likelihood of being selected by the OOM killer within the constrained cgroup. A lower score makes a process less likely to be killed.

In ML training and inference clusters, memory cgroups are used alongside GPU isolation (e.g., MIG, MPS) to enforce host memory quotas for each job or user. This is critical for preventing a single job from causing system-wide swapping or OOM scenarios.

• Protects Critical Daemons: Ensures the Kubernetes kubelet, container runtime, and GPU drivers have guaranteed host memory.
• Complements GPU Memory Limits: While tools like nvidia-smi limit GPU VRAM, host memory for data loading, preprocessing, and model weights is managed via cgroups.
• Enables Fair Sharing: In shared research environments, cgroups prevent one user's memory-intensive data pipeline from starving others.

On resource-constrained embedded Linux devices, memory cgroups are used to protect system services and guarantee resources for critical functions.

• Partitions User Space: Isolates the main application from essential system daemons (e.g., network managers, logging).
• Enforces Hard Limits: Prevents a buggy or malicious user application from consuming all RAM and crashing the entire device.
• Memory Protection Without Virtualization: Provides container-like isolation where running a full container runtime (like Docker) is too heavyweight for the device's resources.

A memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU. This allows both processors to access the same memory pointers using a common page table, simplifying data sharing and enabling techniques like demand paging and zero-copy transfers. It abstracts the physical location of data, allowing pages to migrate between host and device memory transparently.

A technique where the total memory allocated to active GPU workloads exceeds the physical GPU memory (VRAM) capacity. The system relies on host system memory (RAM) or even storage (NVMe SSDs) as a backing store. This is managed by the driver and OS through:

• Page Migration: Moving pages between GPU memory and host memory.
• Demand Paging: Loading data into GPU memory only when accessed (triggering a GPU page fault). This allows for larger working sets but can introduce latency if paging is frequent.

Host (CPU) memory that is prevented from being paged out to disk by the operating system. Key characteristics:

• Enables High-Bandwidth DMA: Allows the GPU to perform Direct Memory Access (DMA) transfers directly to/from this memory, bypassing CPU involvement for copy operations.
• Required for Async Transfers: Essential for overlapping computation and data transfer using CUDA streams.
• System Impact: Overuse can reduce available memory for the OS and other processes, as it is non-pageable. It is a limited resource.

A pre-allocated, large block of memory (on host or device) from which smaller allocations are sub-allocated by the application. This strategy is critical for performance:

• Reduces Allocation Overhead: Avoids frequent, expensive calls to system allocators like cudaMalloc/cudaFree.
• Mitigates Fragmentation: By managing allocations within a contiguous block, it prevents memory fragmentation where free memory is scattered in small, unusable chunks.
• Enables Stream-Ordered Allocation: Allocations can be tied to a CUDA stream's lifecycle for automatic, asynchronous reuse.