Out-of-Memory (OOM) Killer

The OOM Killer activates when the system faces a critical memory shortage and cannot satisfy an allocation request, even after the kernel has exhausted all other memory management techniques. This includes:

• Aggressive swapping to disk (if swap is enabled and not full).
• Dropping clean page caches and reclaimable slab objects.
• Compacting memory.

When these measures fail, the kernel invokes the OOM Killer to free memory by force, preventing a kernel panic.

The kernel selects a victim process using a heuristic badness score (oom_score). This score, visible in /proc/[pid]/oom_score, is calculated based on multiple factors to identify the 'best' process to kill:

• Memory Usage: The primary factor. Processes using more memory, especially resident set size (RSS), receive higher scores.
• Process Lifetime: Long-running daemons and critical system processes are penalized with lower scores.
• User Privileges: Root-owned processes typically have their score adjusted to be less likely targets.
• CPU Time: Processes that have consumed significant CPU time may be penalized.
• OOM Score Adjustment (oom_score_adj): A user-settable value from -1000 to +1000 that directly shifts the final score, allowing administrators to protect critical processes or mark others as preferred victims.

In machine learning inference servers, the OOM Killer poses a significant risk, often terminating the very process managing the GPU. Key failure scenarios include:

• Host Memory Exhaustion: A large language model inference engine, like vLLM or TensorRT-LLM, may allocate substantial host memory for request queues, tokenizers, and communication buffers. If this exhausts system RAM, the kernel may kill the inference server process.
• Indirect GPU Memory Pressure: While the OOM Killer doesn't directly manage GPU memory, system memory is used for GPU driver control structures, pinned memory for DMA, and Unified Memory pages. Exhausting system memory can cripple GPU operations.
• Cascading Failures: Killing a critical orchestration process (e.g., a Kubernetes pod manager) can lead to uncontrolled restarts and service disruption.

System administrators and ML Ops engineers can configure the OOM Killer's behavior to protect critical services:

• oom_score_adj: Set this to -1000 for essential processes (e.g., the inference server, monitoring agent) in a container's spec.containers[].resources.limits.oomScoreAdj or via echo -1000 > /proc/[pid]/oom_score_adj.
• Memory Cgroups (Control Groups): The primary modern defense. By placing workloads in cgroups with strict memory limits (memory.limit_in_bytes), you ensure a single container's memory exhaustion triggers a cgroup-specific OOM kill, isolating the failure and protecting the host system.
• Disabling the OOM Killer: Setting vm.overcommit_memory = 2 and providing ample swap can make the kernel refuse allocations rather than invoke the killer, but this often causes application-level failures instead.
• Monitoring: Tools like dmesg and the system log (/var/log/syslog) capture OOM Killer events, showing the killed process's PID and its oom_score.

The Linux host OOM Killer is distinct from, but interacts with, GPU-level memory management:

• Separate Domains: The OOM Killer operates on system (host) RAM. GPU device memory (VRAM) is managed by the GPU driver and runtime (e.g., CUDA). A GPU-side out-of-memory error (cudaErrorMemoryAllocation) is a different event handled within the application.
• Unified Memory Pressure: When using Unified Virtual Memory (UVM), pages can migrate between host and GPU memory. Severe host memory pressure can stall these migrations, indirectly affecting GPU performance and potentially leading to host OOM conditions.
• Pinned Memory Overhead: Allocating large amounts of page-locked (pinned) host memory for fast GPU transfers reduces the pool of swappable memory for the OS, increasing the risk of triggering the OOM Killer for other processes.

To prevent the OOM Killer from disrupting production model serving:

1. Set Explicit Limits: Always use memory cgroups (via Kubernetes resources.limits.memory or Docker -m) to enforce hard boundaries on container memory usage.
2. Monitor Memory Trends: Use profiling tools to understand the resident set size (RSS) of your inference server, not just its GPU memory allocation.
3. Size Host Memory Appropriately: Ensure the physical host has sufficient RAM for the OS, monitoring tools, and the inference engine's host-side overhead (often 20-30% beyond model weights).
4. Tune Swappiness: Adjust vm.swappiness to a lower value (e.g., 10) to make the kernel prefer caching over swapping, but be aware this can make the OOM Killer activate sooner.
5. Implement Graceful Degradation: Design your inference service to handle memory pressure signals (e.g., listening for SIGKILL or monitoring cgroup pressure) and shed load or queue requests before a forced termination occurs.

A Linux kernel feature that limits, accounts for, and isolates the memory resource usage of a collection of processes. It is the primary mechanism for enforcing memory quotas on containers (e.g., Docker, Kubernetes). The OOM Killer uses cgroup-level memory pressure and over-limit status as critical signals when selecting a victim process to terminate.

• Enforces Hard Limits: Prevents a single container from consuming all host memory.
• Hierarchical Accounting: Memory usage is tracked up a tree of cgroups, allowing for subtree limits.
• Direct Link to OOM: When a cgroup exceeds its limit and cannot reclaim memory, an OOM kill is triggered within that cgroup's scope.

Host (CPU) memory that is prevented from being paged out to disk by the operating system. This is a prerequisite for high-bandwidth Direct Memory Access (DMA) transfers to GPU device memory. Excessive allocation of pinned memory can itself trigger system-wide memory pressure, indirectly leading to OOM Killer activation.

• DMA Requirement: Enables GPUs to directly read/write host memory without CPU intervention.
• Non-Swappable: Cannot be paged to disk, reducing available memory for other processes.
• Performance vs. Risk: Critical for throughput but must be managed carefully to avoid inducing OOM conditions.

A technique where the total memory allocated by processes exceeds the available physical RAM. The kernel allows this, relying on the assumption that not all allocated memory will be used simultaneously. This is common in GPU workloads using Unified Virtual Memory (UVM). When overcommitted memory is actually accessed (demand paging), it can lead to severe swapping and memory pressure, forcing the OOM Killer to intervene.

• Virtual Memory Foundation: Relies on paging (swap) to back excess allocations.
• High-Risk for GPUs: GPU page faults to system swap are extremely slow, causing system-wide stalls.
• OOM Trigger: Aggressive overcommit is a direct precursor to OOM events.

A virtual memory technique where data is transferred from a slower backing store (like system RAM or SSD) into faster GPU memory only when the GPU attempts to access it, signaled by a page fault. In GPU contexts with Unified Virtual Memory, excessive demand paging can create I/O bottlenecks and memory pressure on the host system, contributing to the conditions that activate the OOM Killer.

• On-Access Loading: Memory is loaded lazily, not pre-emptively.
• Page Fault Cost: GPU page faults to host memory or disk are high-latency events.
• Cascading Pressure: Slow paging can cause process stalls, leading to backlog and increased memory residency.

The internal algorithm the Linux OOM Killer uses to select a victim process. It calculates an 'oom_score' for each process based on a 'badness' heuristic. The process with the highest score is killed. Understanding this heuristic is key to influencing the Killer's decisions.

Heuristic Factors Include:

• Memory Usage: Proportional and absolute memory consumption.
• Process Lifetime: Long-running processes may be penalized.
• Process Priority (nice value): Lower priority (higher nice) processes score higher.
• OOM Score Adjustment: Administrators can adjust /proc/[pid]/oom_score_adj to protect or expose specific processes.

A specialized instance of the OOM Killer that operates within the scope of a memory cgroup. When a cgroup hits its hard memory limit and cannot reclaim enough memory, this killer terminates a process within that cgroup instead of a system-wide process. This provides crucial isolation, preventing a rogue container from taking down the entire host.

• Container-Safe: Failure is contained to the offending container/pod.
• Kubernetes Integration: This is the mechanism behind Pod eviction on memory pressure in Kubernetes.
• Policy-Driven: Often integrated with higher-level orchestration policies (e.g., Quality of Service classes for Pods).