Cache Coherence

The most common coherence protocol where a processor writing to a shared memory location invalidates all other cached copies of that data block. Subsequent reads by other processors will miss in their local cache and must fetch the updated value from main memory or the writer's cache. This protocol minimizes write traffic but can cause stalls if multiple processors frequently write to the same location.

An alternative protocol where a write operation broadcasts the new data to update all other cached copies immediately. This reduces read latency for other processors but generates significantly more bus or interconnect traffic on every write. It is less common in performance-critical edge systems due to this bandwidth overhead, which can conflict with high-throughput AI inference data flows.

A precise, widely implemented finite-state machine that defines the state of each cache line. The four states are:

• Modified (M): The line is dirty (different from main memory) and present only in this cache.
• Exclusive (E): The line is clean (matches main memory) and present only in this cache.
• Shared (S): The line is clean and may be present in other caches.
• Invalid (I): The line is not present or is stale. Transitions between these states are triggered by local processor reads/writes and snooped bus transactions from other processors.

A broadcast-based coherence method where all caches monitor (snoop on) a shared bus to see transactions from other processors. When a cache sees a read or write request for an address it holds, it takes action (e.g., invalidates its copy, supplies data). This is simple but does not scale well to many cores due to bus contention. Common in smaller, embedded multi-core processors used in edge AI.

A scalable coherence method that uses a centralized or distributed directory to track which caches hold copies of each memory block. On a write, the system consults the directory to identify and contact only the caches that hold the block, rather than broadcasting. This reduces interconnect traffic but adds directory lookup latency. Used in larger many-core systems and some high-performance edge AI accelerators.

The formal contract between hardware and software that defines the visible ordering of memory operations (reads and writes) across processors. Cache coherence guarantees that writes to the same location become visible in the same order to all processors (single-writer, multiple-reader). However, it does not guarantee the order in which writes to different locations become visible. Strong models like sequential consistency or weaker models like release consistency build upon the coherence foundation to provide these ordering guarantees, which are crucial for correct concurrent programming in multi-threaded AI pipelines.

Memory bandwidth is the maximum rate at which data can be read from or written to a system's memory by the processor. It is a critical bottleneck for data-intensive AI inference. Cache coherence protocols directly impact effective bandwidth by managing the flow of data between cores and main memory.

• High bandwidth is required to feed tensor operations on NPUs/GPUs.
• Coherence traffic (snoops, invalidations) consumes bandwidth, potentially contending with application data.
• Optimizing data locality within caches reduces pressure on the memory bus, improving overall system throughput.

Heterogeneous computing integrates different processing units (CPUs, GPUs, NPUs, DSPs) into a single system-on-chip (SoC), common in edge AI processors. Cache coherence becomes significantly more complex in these environments.

• Different accelerators may have incoherent caches or require explicit software management.
• Unified memory architectures rely on hardware coherence to allow CPUs and accelerators to share data seamlessly.
• System-level performance depends on efficient data movement between heterogeneous cores, governed by coherence protocols.

Deterministic execution ensures a given input always produces the exact same output within a predictable, bounded time. This is paramount for safety-critical edge AI (e.g., robotics, automotive). Cache coherence is a key enabler of determinism in multi-core systems.

• Non-deterministic cache misses or coherence state transitions can cause timing variability.
• Time-predictable architectures often use simpler, analyzable coherence protocols (e.g., time-division multiplexed access) to guarantee Worst-Case Execution Time (WCET).
• Lack of coherence can lead to data races, causing non-deterministic program outputs.

Worst-Case Execution Time (WCET) analysis calculates the maximum possible time a task (like a model inference) can take under any input and system state. Cache coherence directly impacts WCET analysis.

• Pessimistic analysis must account for worst-case coherence overheads (e.g., cache line invalidations from all other cores).
• Analyzable coherence protocols (e.g., deterministic snooping) are preferred for real-time systems to establish safe, tight WCET bounds.
• Unbounded coherence delays make formal timing guarantees impossible, violating requirements for autonomous systems.

Performance isolation prevents the activity of one process or core from degrading the performance of another on shared hardware. Cache coherence mechanisms can be a source of performance interference in multi-tenant edge deployments.

• A core running a noisy background task can generate excessive snoop traffic, slowing down a critical AI inference pipeline on another core.
• Cache partitioning (way masking) and bandwidth allocation techniques are used alongside coherence protocols to enforce isolation.
• Without isolation, SLOs for latency-sensitive AI services cannot be reliably met.

A Real-Time Operating System (RTOS) provides deterministic scheduling and minimal interrupt latency for time-critical applications. The RTOS kernel and its scheduler must be designed to work correctly with the underlying hardware's cache coherence protocol.

• Cache locking may be used by an RTOS to pin critical code/data in cache, avoiding unpredictable misses.
• Synchronization primitives (spinlocks, barriers) rely on atomic operations guaranteed by the coherence protocol.
• The RTOS must manage cache flushes and invalidations when context switching between tasks to maintain correctness.