Backpressure

Backpressure creates a feedback loop between data producers and consumers. When a consumer's buffer or processing queue exceeds a defined threshold, it emits a signal upstream to slow or pause transmission.

• Push-based systems require explicit backpressure mechanisms to avoid overwhelming consumers
• Pull-based systems (like Reactive Streams) naturally implement backpressure by having consumers request data in chunks
• Common in stream processing engines like Apache Kafka, where consumers control their own consumption rate via offset commits

Without backpressure, unbounded buffers grow until they exhaust available memory, causing OutOfMemoryError crashes or severe garbage collection pauses.

• A bounded buffer with a fixed capacity is the simplest backpressure mechanism—producers block when the buffer is full
• TCP flow control uses a sliding window protocol where the receiver advertises its available buffer space to the sender
• In LLM inference pipelines, backpressure prevents embedding generation queues from accumulating when the vector store ingestion rate falls behind

The Reactive Streams initiative defines a standard for asynchronous stream processing with non-blocking backpressure across JVM languages.

• Core interfaces: Publisher, Subscriber, Subscription, and Processor
• The Subscriber calls subscription.request(n) to pull exactly n items, preventing the Publisher from pushing more than the consumer can handle
• Implementations include Project Reactor, RxJava, and Akka Streams, all interoperable via the java.util.concurrent.Flow API in Java 9+

When backpressure signals are ignored or propagation is too slow, systems must choose between load shedding and catastrophic failure.

• Load shedding intentionally drops excess requests (e.g., returning HTTP 503) to protect system integrity
• Backpressure preserves all data by slowing the pipeline, but may increase end-to-end latency
• Adaptive strategies combine both: apply backpressure up to a timeout threshold, then shed load to maintain SLO compliance for P99 latency targets

In Answer Engine Architectures, backpressure governs the flow between retrieval stages to prevent cascading timeouts.

• A semantic cache that experiences a cache stampede can signal the embedding service to throttle new vectorization requests
• gRPC streaming connections between the orchestrator and re-ranker use HTTP/2 flow control windows to apply backpressure at the transport layer
• When ANN search latency spikes due to index compaction, backpressure on the query planner prevents building up a backlog of unprocessed requests that would violate the Time-to-First-Token budget

Backpressure mechanisms often work in tandem with circuit breakers to handle scenarios where downstream services are completely unresponsive rather than merely slow.

• A circuit breaker trips open after a configurable failure threshold, immediately rejecting requests without waiting for timeouts
• In the half-open state, the circuit breaker probes the downstream service with limited traffic to test recovery
• Combined with backpressure, this prevents retry storms where overwhelmed services receive duplicate requests from impatient upstream clients, compounding the overload