Garbage Collection Pause

During a stop-the-world pause, all mutator (application) threads are halted. The garbage collector then executes a cycle—typically marking live objects, sweeping dead ones, and compacting memory. Root scanning (analyzing global variables and thread stacks) is a key contributor to pause time. The duration is a function of the live data set size, not just total heap size.

Most modern collectors are generational, segregating objects by age into a Young Generation (Eden/Survivor spaces) and an Old Generation (Tenured). Minor GC events in the Young Generation are typically fast (sub-millisecond) because most objects die young. Major or Full GC events compact the Old Generation and can cause multi-second pauses.

A GC pause cannot start arbitrarily. Threads must reach a safepoint—a specific instruction boundary where the runtime has precise knowledge of the stack layout. A common latency bug is safepoint biasing, where a thread executing a long counted loop or JNI call delays reaching a safepoint, extending the pause for all other threads.

There is a fundamental trade-off. Throughput collectors (e.g., Parallel GC) maximize CPU utilization for batch processing but incur long pauses. Low-pause collectors (e.g., ZGC, Shenandoah) perform concurrent compaction to achieve sub-millisecond pauses, but they consume more CPU cycles and reduce peak application throughput.

Modern concurrent collectors like ZGC and Shenandoah use colored pointers or forwarding tables to move objects while the application is running. A brief final pause (often < 1ms) is still required to drain the work queue and update root references. This decouples pause time from heap size, making terabyte-scale heaps viable.

Key metrics for diagnosing pause issues include:

• gc.pause.time: Total stop-the-world duration.
• gc.frequency: How often collections trigger.
• allocation rate: High allocation pressure fills the Young Generation faster.
• promotion rate: Objects prematurely promoted to the Old Generation cause fragmentation and costly Full GCs.

A phase during garbage collection where all application threads are suspended to allow the collector to safely reclaim memory. In real-time AI inference, even a 10ms pause can cause a tail latency spike that violates a service-level objective (SLO). Modern collectors use incremental or concurrent strategies to minimize this duration.

The empirical observation that most objects die young. Generational collectors segregate objects by age, scavenging the 'young generation' frequently and promoting survivors to an 'old generation'. This exploits the high infant mortality rate of temporary objects in AI feature engineering loops, dramatically reducing average pause time.

The foundational algorithm for tracing live objects. Objects are classified as white (unreachable), gray (reachable but children unscanned), or black (reachable and scanned). The invariant—no black object points to a white object—is maintained by a write barrier during concurrent marking, preventing the collector from missing live references.

An alternative to tracing collectors where memory is allocated in large, deallocated en masse arenas or regions. This is highly effective for AI inference servers processing discrete requests, as all temporary tensors and activations for a single prediction can be freed instantly without a global pause, achieving near-zero collection overhead.

A technique to bypass the garbage collector entirely by allocating objects directly on the stack or inline within other objects. Languages like C# (structs) and Java (Project Valhalla) use this for small, immutable data like vector coordinates. This eliminates heap pressure and pointer indirection, critical for high-throughput AI data pipelines.

Advanced algorithms like ZGC and Shenandoah aim for sub-millisecond max pause times by performing almost all work—marking, relocation, and remapping—concurrently with application threads. They use load barriers and colored pointers to ensure the mutator never sees an inconsistent object graph, enabling Java-based AI serving with predictable low latency.