Cache Coherency

This is the most common coherence protocol. When a processor writes to a shared cache line, it invalidates all other copies in the system. Subsequent reads by other processors will cause a cache miss, forcing them to fetch the updated data from the writer's cache or main memory.

• Core Mechanism: A write operation broadcasts an invalidation signal.
• State Transition: Other caches transition the line to an Invalid state.
• Benefit: Simple and bandwidth-efficient for write-heavy, non-shared data.
• Drawback: Can cause performance penalties if other processors frequently need to read the same data shortly after a write.

Under this protocol, when a processor writes to a shared cache line, it broadcasts the new data to all other caches holding a copy, updating them immediately.

• Core Mechanism: Write operations broadcast the new data value.
• State Maintenance: Other caches update their copy and may remain in a Shared state.
• Benefit: Subsequent reads by other processors are fast hits, reducing latency for read-heavy shared data.
• Drawback: Consumes significant interconnect bandwidth with every write to shared data, which can be prohibitive in many-core systems like modern GPUs.

MESI is a specific, widely implemented write-invalidate protocol where each cache line can be in one of four states:

• Modified (M): The line is dirty (modified) and exists only in this cache. The cache must write it back to memory before it can be used elsewhere.
• Exclusive (E): The line is clean (matches main memory) and exists only in this cache. The processor can write to it without notifying others, transitioning it to Modified.
• Shared (S): The line is clean and may be present in other caches. It is read-only.
• Invalid (I): The line is not present or is stale and cannot be used.

These states are maintained through snooping or directory-based protocols to track ownership and sharing.

These are the two primary architectural approaches to implementing coherence protocols.

Snooping (Bus-Based):

• All caches monitor (snoop) a shared broadcast interconnect (bus) for transactions.
• Upon seeing a relevant transaction (e.g., a write), a cache takes action (e.g., invalidates its copy).
• Pros: Simple, low latency for small-scale systems.
• Cons: Broadcast traffic doesn't scale well to many processors (e.g., large multi-GPU systems).

Directory-Based:

• A central directory keeps track of which caches have copies of each memory block.
• On a write, the directory is consulted and point-to-point messages are sent only to the caches that hold a copy.
• Pros: Scales efficiently to large processor counts, used in modern NUMA and multi-GPU systems.
• Cons: Adds directory lookup latency and requires dedicated storage.

False sharing is a critical performance pitfall in coherent systems. It occurs when two unrelated variables reside on the same cache line, and different processors modify these independent variables.

• Mechanism: Processor A writes to Variable X. Processor B writes to unrelated Variable Y, which is on the same cache line. The coherence protocol treats the entire line as shared data, triggering invalidations back and forth.
• Symptom: High, unnecessary coherence traffic and cache misses, despite no actual data dependency.
• Solution: Data structure padding to ensure independent variables land on separate cache lines, or careful data layout in GPU kernels to align access patterns with warp boundaries.

Maintaining coherence between CPU and GPU caches is complex due to their separate memory hierarchies. Modern systems use several approaches:

• Unified Virtual Memory (UVM) with Atomics: Systems like CUDA UVM provide a unified address space. Coherence is often managed at page granularity using GPU page faults and migration. Atomic operations across CPU and GPU may trigger fine-grained coherence actions.
• Explicit Management: Lower-level APIs (e.g., CUDA) often leave coherence management to the programmer using explicit memory barriers (__threadfence(), __syncthreads()) and stream synchronization to ensure visibility of writes.
• Hardware Support: Technologies like NVIDIA's NVLink with atomic operations can enable cache coherence between CPU and GPU memory, allowing them to operate on a shared, coherent memory pool with reduced programmer burden.

Cache coherency is a hardware-level protocol that maintains data consistency across multiple processor caches accessing the same memory location. The primary goal is to ensure that any read of a memory address returns the most recently written value, regardless of which processor performed the write.

• MESI Protocol: The most common protocol defines four states for each cache line: Modified, Exclusive, Shared, and Invalid. State transitions are coordinated via snooping or directory-based messages across the system interconnect.
• Snooping: Each cache controller monitors (snoops) the shared bus for transactions from other caches and invalidates or updates its local copies accordingly.
• Directory-Based: A central directory tracks which caches hold copies of each memory block, reducing broadcast traffic in large-scale systems.

In distributed training across multiple GPUs, cache coherency is managed at the system level to synchronize model parameters and gradients. Inconsistent caches can lead to stale weight updates and training divergence.

• Parameter Server Architectures: The server maintains the master copy of parameters. Worker GPUs cache parameters locally; coherency protocols ensure workers fetch the latest version after each synchronization step.
• All-Reduce Operations: Collective operations used in data-parallel training (e.g., NCCL) implicitly handle coherency by ensuring all GPUs have a consistent view of aggregated gradients before the next forward pass.
• NVLink with Cache Coherency: NVIDIA's NVLink interconnect for GPUs can support cache-coherent access, allowing GPUs to directly access each other's memory with automatic coherence, simplifying programming models for multi-GPU systems.

Modern AI systems often feature tight integration between CPUs and GPUs, requiring cache coherency across these different processor types to avoid explicit data synchronization calls.

• Unified Virtual Memory (UVM): Systems like CUDA UVM create a single virtual address space shared by CPU and GPU. Hardware cache coherency (e.g., on NVIDIA Grace Hopper Superchips) allows both processors to access the same data seamlessly. The GPU can access CPU-side data structures (like hash tables for tokenizers) without manual cudaMemcpy operations.
• Demand Paging & Migration: When a GPU accesses a page not in its memory, a page fault occurs. The memory management unit (MMU) can migrate the page from CPU memory to GPU memory, maintaining coherence throughout.
• Reduced Programming Complexity: Coherent systems allow developers to use pointer-based data structures across devices, moving away from the traditional discrete copy model.

It is crucial to distinguish between cache coherency and memory consistency. Coherency defines the behavior of reads and writes to a single memory location. Consistency defines the observable order of reads and writes to multiple locations across threads.

• Sequential Consistency: The strongest model. The result of any execution appears as if all operations were executed in some sequential order consistent with program order.
• Weak Consistency Models: Used in GPUs (e.g., CUDA's memory model) for performance. Writes to global memory are not immediately visible to other threads without explicit memory fences (__threadfence()) or synchronization.
• Implication for AI: Kernel launches and asynchronous operations must use proper fences and synchronization (e.g., cudaStreamSynchronize) to ensure that updated parameters or intermediate activations are visible to subsequent kernels, even if cache coherency is maintained.

Maintaining strict cache coherency introduces overhead that can become a bottleneck at scale. System architects make deliberate trade-offs between coherence, performance, and complexity.

• Coherence Traffic Overhead: Every write to a shared cache line may generate invalidation messages to all other caches holding that line, consuming interconnect bandwidth.
• False Sharing: Occurs when unrelated variables reside on the same cache line. Updates by one processor invalidate the entire line for others, causing unnecessary coherence traffic and performance degradation. This is a common issue in multi-threaded data loaders.
• Scalability Limits: Snooping protocols broadcast on a shared bus, which does not scale beyond dozens of processors. Large-scale AI clusters (hundreds of GPUs) often use relaxed consistency models or software-managed coherence for shared parameters to avoid this overhead.

Recent hardware advancements are designed to provide efficient cache coherency specifically for AI workloads, which have predictable access patterns.

• NVIDIA Grace Hopper: Features a coherent NVLink-C2C chip-to-chip interconnect between the Grace CPU and Hopper GPU, enabling a single, coherent memory space. This drastically reduces latency for CPU-GPU data sharing in inference pipelines.
• AMD Infinity Fabric: Provides coherent links between AMD CPUs and GPUs (e.g., MI300A APU), allowing unified memory programming for heterogeneous computing.
• Domain-Specific Coherence: For transformer inference, the KV cache is a large, read-heavy data structure shared across decoding steps. Hardware or software can optimize coherence for this pattern, treating it as mostly read-only to minimize invalidations.

A memory consistency model defines the legal ordering of memory operations (reads and writes) as observed by multiple processors or threads. It provides the formal rules that hardware and software must follow. Cache coherency is a subset of consistency, guaranteeing that writes to the same location are serialized. Stronger models (like sequential consistency) provide stricter guarantees but may impact performance, while weaker models (like release consistency) allow for more hardware optimizations.

A snooping protocol is a method for maintaining cache coherency in a bus-based multiprocessor system. Each cache controller continuously "snoops" on the shared memory bus, monitoring transactions from other processors. When it sees a write to a memory address it has cached, it either invalidates or updates its own copy. This is a broadcast-based approach, contrasting with directory-based protocols used in larger, non-bus systems.

Directory-based coherency is a scalable alternative to snooping for large multiprocessor or multi-GPU systems. A central directory maintains the state (shared, modified) of each memory block and tracks which caches hold copies. On a write, the directory sends point-to-point invalidation messages only to the caches that have the block, avoiding broadcast traffic. This is essential for systems with non-uniform interconnect architectures like NVLink or Infinity Fabric.

These are two fundamental cache write policies that interact directly with coherency protocols:

• Write-Back: Data is written only to the cache. The modified line is written back to main memory only when it is evicted. This reduces memory traffic but requires coherency protocols to track the "Modified" (M) state.
• Write-Through: Every write updates both the cache and main memory immediately. This simplifies coherency but creates high bandwidth demand. Most modern high-performance systems use write-back caches for efficiency, relying on robust protocols like MESI to manage consistency.

False sharing is a performance degradation that occurs when two unrelated variables reside on the same cache line, and different processors modify them independently. Although the variables are logically separate, the cache coherency protocol treats the entire cache line as a single unit. This causes unnecessary invalidation and movement of the line between caches, leading to cache thrashing and reduced performance. It is a critical consideration for parallel programming and data structure design.