gRPC Streaming

A standard unary RPC follows a strict request-response cycle: one client message, one server reply, connection closed. Server-side streaming RPC modifies this by allowing the server to send a stream of messages after receiving a single client request. The client reads until the stream ends. This is the foundational pattern for real-time token delivery in LLM inference, where a single prompt yields hundreds of generated tokens sent incrementally.

• Unary: rpc GetAnswer(Query) returns (Answer);
• Server Streaming: rpc StreamTokens(Query) returns (stream Token);
• Bidirectional: rpc Chat(stream Message) returns (stream Message);

gRPC streaming is built on HTTP/2's multiplexing capability, which allows multiple concurrent streams over a single TCP connection without head-of-line blocking. Each stream is an independent, bidirectional sequence of frames. This means a single gRPC channel can handle hundreds of parallel streaming RPCs simultaneously, each delivering token sequences to different users, without the connection setup overhead of HTTP/1.1 keep-alive pools.

• Frame-based: Messages are serialized into binary frames.
• Stream IDs: Each logical stream gets a unique identifier.
• Flow Control: Per-stream and connection-level flow control prevent buffer bloat.

gRPC uses Protocol Buffers (protobuf) as its Interface Definition Language (IDL) and wire format. Service methods and streaming payloads are defined in .proto files. Protobuf's binary serialization is significantly more compact than JSON, reducing the byte overhead per token delivered. For a token stream, each Token message is serialized to a minimal binary frame, keeping per-token latency low.

protobufCopy
service LLMService {
  rpc Generate(GenerateRequest) returns (stream Token);
}
message Token {
  string text = 1;
  int32 index = 2;
}

HTTP/2 provides credit-based flow control at the stream and connection level. The receiver advertises how many bytes it is willing to accept. If a client consuming a token stream is slow—perhaps rendering UI updates—it can reduce its window size. The gRPC server's send operations block or buffer until the client grants more credit. This prevents overwhelming slow consumers and avoids unbounded memory growth on the server.

• Stream-level window: Controls individual RPC throughput.
• Connection-level window: Aggregates all streams on a single TCP connection.
• Automatic negotiation: Handled by gRPC runtime, not application code.

Every gRPC call can specify a deadline—an absolute point in time by which the RPC must complete. For streaming RPCs, the deadline applies to the entire stream lifetime. If a client disconnects or a deadline is exceeded, gRPC propagates a cancellation signal to the server, which can immediately halt token generation and free GPU resources. This is critical for infrastructure cost control in LLM serving, where abandoned requests waste compute.

• Deadline propagation: Sent in gRPC metadata headers.
• Cancellation: Triggers context.Canceled on the server side.
• Resource reclamation: Stops model inference immediately.

gRPC provides interceptors—client-side and server-side middleware that can observe and modify RPCs. For streaming calls, interceptors can wrap the stream to log each token, inject authentication metadata, collect per-token latency metrics, or implement retry logic. This enables observability without modifying service logic.

• Unary interceptor: UnaryServerInterceptor
• Stream interceptor: StreamServerInterceptor wraps ServerStream
• Use cases: Auth injection, logging, tracing, rate limiting

The critical metric measuring the delay between a user's query submission and the generation of the very first response token. In a gRPC streaming context, a low TTFT is achieved by immediately sending the first token over the established stream, rather than waiting for the entire response to be generated. This directly combats the perception of latency. A poor TTFT, often caused by slow model inference or network handshakes, makes a system feel sluggish even if subsequent token generation is fast.

Represents the worst-case response time for 99% of requests, capturing the outlier performance that defines a frustrating user experience. While gRPC streaming provides a low-latency median experience, tail latency can spike due to garbage collection pauses, network congestion, or model queuing. Engineering for the P99 involves optimizing the entire pipeline, from connection pooling to continuous batching, to ensure the streaming connection remains responsive even under peak load.

A vital flow control mechanism in distributed streaming systems. If a gRPC server is generating tokens faster than a slow client can process them, the client's buffers risk overflow. gRPC streaming leverages HTTP/2 flow control to implement backpressure, allowing the client to signal the server to pause transmission. This prevents data loss and system crashes, ensuring a resilient, end-to-end data pipeline without overwhelming any single component.

A dynamic inference optimization technique that is a natural complement to gRPC streaming. Instead of waiting for an entire batch of requests to complete, the scheduler appends new sequences to the running batch as soon as previous ones finish. This maximizes GPU utilization and dramatically increases throughput. When combined with streaming, the generated tokens from these continuously updated batches can be pushed to clients immediately, minimizing both server cost and client-perceived latency.

A core memory mechanism within transformer models that stores the computed Key and Value tensors from previously generated tokens. During gRPC streaming for text generation, the KV-cache eliminates redundant computation for the entire sequence history on each new token generation step. This is the primary reason autoregressive generation is fast enough to stream token-by-token. Without an efficient KV-cache, the latency between streamed tokens would be prohibitively high.

A resilience design pattern where a system maintains partial core functionality when a dependency fails. In a gRPC streaming architecture, if a secondary service like a fact-checking re-ranker becomes unavailable, the system can degrade gracefully by continuing to stream the base model's response with a warning, rather than breaking the stream entirely. This is often implemented with a circuit breaker pattern to detect the failing dependency and switch to a fallback path.