Lock-Free Programming

The defining property of lock-free algorithms: if any thread is suspended or fails, the remaining threads continue to make forward progress. This is achieved through atomic primitives like compare-and-swap (CAS) rather than mutual exclusion locks.

• Prevents deadlock entirely—no thread holds a resource while waiting for another
• Eliminates priority inversion, where a high-priority thread is blocked by a low-priority one
• Critical for real-time AI inference where latency spikes from lock contention are unacceptable

The fundamental building block of lock-free data structures. CAS atomically compares a memory location's value with an expected value and, if they match, swaps in a new value. This single indivisible hardware instruction enables threads to safely update shared state without locks.

• Implemented via CPU instructions like CMPXCHG on x86 or LDREX/STREX on ARM
• Forms the basis for lock-free stacks, queues, and hash tables
• The ABA problem—where a value changes from A to B and back to A—is a classic CAS pitfall requiring tagged pointers

Lock-free sits on a spectrum of non-blocking guarantees. Wait-free algorithms guarantee every thread completes in a bounded number of steps—the strongest guarantee but hardest to implement. Lock-free guarantees system-wide progress but individual threads may starve. Obstruction-free is the weakest: a thread makes progress only when running in isolation.

• Wait-free: Used in hard real-time systems (e.g., avionics, pacemakers)
• Lock-free: The pragmatic sweet spot for high-throughput AI model serving
• Obstruction-free: Rarely used alone; often a stepping stone to lock-free designs

Lock-free algorithms face a unique challenge: safely freeing memory when no thread holds a reference. Hazard pointers let threads publish which objects they're accessing, preventing premature deallocation. Epoch-based reclamation (EBR) groups operations into time epochs and recycles memory only when all threads have left an epoch.

• Hazard pointers: Per-thread lists of 'hazardous' references, checked before freeing
• EBR: Lower overhead but can delay reclamation indefinitely if a thread stalls
• Both are essential for production lock-free data structures in languages without garbage collection

Multi-producer, multi-consumer (MPMC) lock-free queues are the backbone of continuous batching in LLM serving engines like vLLM. Requests arrive asynchronously and are batched dynamically without blocking inference threads.

• Michael-Scott queue: A classic lock-free FIFO using CAS on head and tail pointers
• Used in NVIDIA Triton Inference Server for request scheduling
• Enables dynamic batching: new requests join a running batch without pausing generation
• Reduces Time to First Token (TTFT) by eliminating lock-induced queuing delays

When multiple threads repeatedly fail CAS operations, contention can degrade performance below locking approaches. Exponential backoff—progressively increasing wait times between retries—reduces cache coherency traffic and improves throughput under high load.

• Without backoff: CAS failures cause cache line bouncing between CPU cores
• With backoff: Threads pause briefly, reducing interconnect pressure
• Advanced strategies include adaptive backoff that adjusts delays based on observed contention levels

An atomic hardware instruction that is the fundamental building block of lock-free programming. CAS updates a memory location only if its current value matches an expected value, returning a boolean success indicator.

• Atomicity: Guaranteed by the CPU, not the OS scheduler
• ABA Problem: A classic vulnerability where a value changes from A to B and back to A, fooling CAS. Solved with tagged pointers or hazard pointers
• x86 Implementation: The CMPXCHG instruction with a LOCK prefix
• ARM Equivalent: Load-Link/Store-Conditional (LDREX/STREX) instructions

The rules governing the visibility of memory writes across CPU cores. Lock-free code must explicitly control ordering to prevent subtle data races that compilers and CPUs can introduce.

• Sequential Consistency: The strongest guarantee; all operations appear in a single total order
• Acquire-Release Semantics: A lighter-weight model where loads synchronize with stores
• Memory Fence: A barrier instruction (MFENCE, DMB) that prevents reordering
• Out-of-Thin-Air Values: A theoretical hazard in relaxed memory models where speculative execution creates impossible values

The Michael & Scott queue is the canonical lock-free FIFO data structure, using a linked list with atomic head and tail pointer updates. Critical for AI inference serving where multiple producer threads enqueue requests and consumer threads batch them.

• Enqueue: CAS on the tail node's next pointer, then swing tail forward
• Dequeue: CAS on the head pointer; if head equals tail, the queue is empty
• Sentinel Node: A dummy node simplifies edge cases when head and tail coincide
• Backoff Strategy: Exponential backoff reduces contention when CAS fails repeatedly

A safe memory reclamation scheme for lock-free data structures that prevents a thread from freeing a node while another thread is still reading it. Each thread publishes a list of hazard pointers identifying nodes it is currently accessing.

• Retirement List: Deleted nodes are placed on a private list, not immediately freed
• Scan Phase: Periodically scan all threads' hazard pointers; only free nodes not referenced by any thread
• ABA Prevention: Hazard pointers also prevent the ABA problem by ensuring a node's memory is not reused while a CAS is in flight
• Alternative: Epoch-Based Reclamation (EBR) offers higher throughput but can stall reclamation indefinitely

A critical distinction in non-blocking progress guarantees. Lock-freedom ensures system-wide progress (at least one thread makes progress), while wait-freedom guarantees per-thread progress (every thread completes in a bounded number of steps).

• Lock-Free: Starvation is possible for individual threads, but the system never deadlocks
• Wait-Free: The gold standard; no thread can be starved, essential for hard real-time AI systems
• Obstruction-Free: The weakest guarantee; a thread completes only if it runs in isolation
• Universal Constructions: Techniques like Herlihy's methodology can transform any sequential object into a wait-free one using CAS and helping

A performance killer in lock-free structures where threads modify independent variables that reside on the same cache line (typically 64 bytes). The cache coherence protocol forces constant invalidation, destroying scalability.

• Cache Line Padding: Inserting unused bytes to ensure hot fields occupy separate cache lines
• MESI Protocol: The state machine (Modified, Exclusive, Shared, Invalid) governing cache line ownership
• @Contended Annotation: JVM's built-in mechanism for automatic padding
• Detection: Use perf c2c on Linux to identify false sharing hotspots in production AI serving pipelines