Sticky Session

Sticky sessions enable the backend application to rely on local state locality. Because all requests from a single session hit the same server, the application can store the full conversational context—including the KV-Cache, chat history, and intermediate reasoning steps—directly in the server's RAM or local ephemeral storage. This avoids the serialization/deserialization overhead and network latency associated with fetching state from a distributed session store like Redis on every API call. For large context windows containing millions of tokens, this local access pattern is critical for maintaining low-latency, real-time interaction speeds.

The primary architectural risk is the orphaned session problem. If the pinned backend server crashes, scales in, or is terminated by an orchestrator, the local state is irretrievably lost. The load balancer routes the next request to a new, cold server that has no knowledge of the prior conversation, resulting in a context collapse from the user's perspective. Mitigation strategies include session replication (copying state to a peer) or implementing a hybrid model where the local cache is backed by a persistent distributed store, allowing a new server to reconstitute the session if the original fails.

Sticky sessions directly conflict with even load distribution. A single server may become a hotspot if it is pinned to a user executing a computationally expensive, long-running agentic task (e.g., multi-hop reasoning over a large codebase). The load balancer cannot redistribute this user's requests to idle servers, leading to resource contention and increased latency for other users pinned to the same host. This requires careful over-provisioning or implementing a least-connections algorithm variant that still respects cookie affinity but accounts for server load during the initial binding.

A modern alternative to sticky sessions is stateless token rehydration. Instead of pinning a user to a server, the entire conversational state is compressed and encoded into a JWT (JSON Web Token) or stored in a fast external cache. The client sends this token with every request, allowing any stateless backend instance to reconstruct the context. This approach enables true horizontal scaling and zero-downtime deployments but introduces a payload size penalty and requires cryptographic verification to prevent context poisoning attacks where a malicious client manipulates the serialized history.

For streaming AI responses, sticky sessions are often mandatory at the transport level. When a client upgrades an HTTP connection to a WebSocket, a persistent, stateful tunnel is established between the client and a specific server. The load balancer must ensure all frames for that connection are routed to the same backend. If the connection is interrupted, the client must establish a new WebSocket, and the application must implement a resume protocol to replay the lost context window from a persistent log, as the new server will not have the original socket's memory.

The persistent data structure that maintains the history, variables, and user intent across multiple turns of a conversation. Without sticky sessions, this state must be externalized to a distributed session store like Redis to survive server hops.

• Stores dialogue history and slot-filled parameters
• Enables resumption of interrupted workflows
• Critical for transactional multi-turn operations

A memory optimization technique that stores the Key and Value tensors of previous tokens to avoid recomputing them during autoregressive text generation. Sticky sessions ensure this cache remains local to the GPU that generated it, avoiding expensive data transfers.

• Reduces latency for subsequent turns by up to 90%
• Memory-bound; cache size grows linearly with context length
• Invalidated entirely if a request routes to a different server

The maximum span of tokens a language model can attend to when generating a response, defining the boundary of its immediate working memory. Sticky sessions allow the accumulation of dialogue history within this window without re-transmission.

• Typical limits: 4K, 32K, 128K, or 1M+ tokens
• Exceeding the limit triggers context window truncation
• The entire window must be present on the serving node

A failure state where the model loses the distinction between different conversational threads or temporal states, flattening the dialogue into a single incoherent prompt. Sticky sessions mitigate this by maintaining a single, linear history on one node.

• Often caused by parallel conversation branching
• Results in the model confusing user identities or intents
• Prevented by strict session isolation and affinity