Server-Sent Events (SSE)

SSE establishes a one-way communication channel from server to client. Once the client initiates the connection via the EventSource API, the server can push data at will, but the client communicates back only through separate HTTP requests. This asymmetry is ideal for token streaming from LLMs, where the model generates output sequentially and the client only needs to display results. Unlike WebSockets, there is no overhead for maintaining a bidirectional protocol when only server-to-client updates are needed.

SSE is built directly into modern browsers through the EventSource API, requiring no external libraries or custom handshake logic. The browser automatically handles:

• Connection establishment via standard HTTP
• Automatic reconnection with exponential backoff on connection loss
• Event parsing and dispatching to JavaScript handlers
• Tracking the last received event ID via the Last-Event-ID header

This native integration simplifies client-side code dramatically compared to WebSocket implementations, which require manual reconnection logic and frame parsing.

A critical feature for production resilience: SSE clients automatically attempt to reconnect when the connection drops. The EventSource API implements an exponential backoff strategy out of the box. The server can also include an id field on each event, allowing the client to send the Last-Event-ID HTTP header upon reconnection. This enables the server to resume the stream from the exact point of interruption, preventing duplicate or missed tokens during LLM generation—a capability essential for maintaining coherent output in long-running inference sessions.

SSE transmits data as UTF-8 encoded text using a simple field-based format. Each message can include:

• event: A named event type for dispatching to specific handlers
• data: The payload, which can span multiple lines
• id: A unique identifier for resumption tracking
• retry: A server-specified reconnection interval in milliseconds

This structure allows a single SSE stream to carry multiple logical event types—for example, streaming token events alongside metadata and error events—without multiplexing overhead. The Content-Type: text/event-stream header signals the protocol to intermediaries.

When served over HTTP/2, SSE connections benefit from multiplexed streams within a single TCP connection. This eliminates the browser's per-domain connection limit problem that plagued HTTP/1.1 SSE implementations, where only 6 concurrent connections were allowed. With HTTP/2, a client can maintain multiple SSE streams to different endpoints simultaneously without head-of-line blocking. The server can also leverage HPACK header compression to reduce the overhead of repeated response headers across long-lived connections.

Because SSE operates over standard HTTP on port 80 or 443, it traverses corporate firewalls, load balancers, and reverse proxies without special configuration. Unlike WebSockets, which require an HTTP Upgrade handshake that some intermediaries may block or mishandle, SSE looks like a regular long-lived HTTP response. This makes SSE the preferred choice for enterprise deployments where network infrastructure cannot be modified. Standard HTTP caching rules apply, and intermediaries can buffer or transform the stream as needed.

An advanced iteration of dynamic batching for generative models that evicts completed sequences from a batch and inserts new requests immediately, preventing idle compute time.

• Eliminates the straggler problem where a single long generation holds up the entire batch
• Critical for maintaining high GPU utilization during streaming token generation via SSE
• Implemented in modern serving engines like vLLM and NVIDIA Triton

A memory mechanism in transformer-based generative models that stores previously computed key and value tensors to avoid redundant computation during autoregressive token generation.

• Each new token requires attending to all prior tokens in the sequence
• Without caching, the computational cost grows quadratically with sequence length
• Efficient KV cache memory allocation is essential for serving many concurrent SSE streams without exhausting GPU memory

A flow control mechanism that signals upstream clients to slow down request rates when a serving system is saturated, preventing queue overflow and cascading failures.

• Essential when thousands of concurrent SSE connections stream tokens simultaneously
• Prevents the server from accepting more work than it can process within latency SLOs
• Works in tandem with load shedding to maintain system stability under peak load

A high-performance, open-source remote procedure call framework that uses Protocol Buffers for serialization and HTTP/2 for transport, commonly used for low-latency streaming inference connections.

• HTTP/2 multiplexing allows multiple concurrent streams over a single TCP connection
• Bidirectional streaming enables richer interaction patterns beyond unidirectional SSE
• Often preferred for server-to-server model serving where strict schema contracts are required

A latency-reduction technique where a small, fast draft model predicts multiple future tokens that are then verified in parallel by a larger target model, accelerating text generation.

• The draft model runs on the client or a lightweight sidecar process
• The target model verifies entire token sequences in a single forward pass
• Dramatically reduces time-to-first-token and overall generation latency for SSE-delivered responses

A deployment strategy where a new model version is rolled out to a small subset of production traffic to validate performance and stability before a full-scale rollout.

• A small percentage of SSE connections receive tokens from the new model version
• Engineers monitor P99 latency and error rates on the canary before promoting it
• Enables safe experimentation with new serving optimizations without risking all users