gRPC

gRPC uses Protocol Buffers (protobuf) as its Interface Definition Language (IDL) and wire-format. This provides:

• Strongly-typed contracts: Service interfaces and message structures are defined in .proto files, generating client and server code.
• Compact binary format: Payloads are significantly smaller than JSON or XML, reducing network I/O and serialization/deserialization CPU overhead.
• Schema evolution: Supports backward and forward compatibility through field numbering, enabling safe API changes without breaking deployed clients.

gRPC is built on HTTP/2, which provides fundamental performance advantages over HTTP/1.1:

• Multiplexed streams: Multiple concurrent requests and responses are interleaved over a single, long-lived TCP connection, eliminating head-of-line blocking.
• Binary framing: HTTP/2 frames headers and data in a binary format, reducing parsing complexity and errors.
• Server push: The server can proactively send resources to the client, though this is less commonly used in standard RPC patterns.
• Flow control: Per-stream and connection-level flow control prevents a fast sender from overwhelming a slow receiver.

gRPC supports four distinct service method types, going beyond simple request-response:

• Unary RPC: A single request from the client and a single response from the server.
• Server streaming RPC: The client sends a request and receives a stream of responses, ideal for delivering a sequence of generated tokens from an LLM.
• Client streaming RPC: The client sends a stream of data (e.g., sensor readings) and receives a single response from the server.
• Bidirectional streaming RPC: Both sides send a sequence of messages independently, enabling full-duplex, real-time communication for use cases like conversational AI.

The protoc compiler generates idiomatic client and server stubs in over 10 languages, including C++, Java, Go, and Python. This provides:

• Type-safe APIs: Generated code eliminates manual request marshaling and provides compile-time guarantees.
• High-performance stubs: The generated serialization and networking code is optimized, avoiding the overhead of reflection-based libraries.
• Consistent cross-platform contracts: A single .proto definition ensures that a Go server and a Python client communicate with identical data structures and service definitions.

gRPC clients can specify a deadline—a point in time by which an RPC must complete. If the deadline is exceeded, the RPC is terminated with a DEADLINE_EXCEEDED error. This is critical for latency-sensitive inference:

• Resource reclamation: Server-side work for a cancelled request is immediately halted, preventing wasted GPU cycles on a response the client will never receive.
• Cascading propagation: Deadlines and cancellation signals are propagated across a chain of dependent microservices, ensuring end-to-end timeout enforcement.
• Predictable tail latency: Combined with circuit breakers, deadlines prevent a slow model replica from causing unbounded queueing delays.

gRPC provides extensible interception points for cross-cutting concerns:

• Authentication: Supports TLS/SSL for transport security, token-based auth (JWT, OAuth2) via metadata, and custom authentication interceptors.
• Load balancing: Client-side and proxy-based load balancing policies (e.g., round-robin, pick-first) are available. A custom resolver can integrate with service discovery systems like etcd or Consul.
• Interceptors: Unary and streaming interceptors allow for logging, monitoring, retry logic, and tracing (e.g., OpenTelemetry) to be injected without modifying business logic.

The interface definition language and binary serialization format that powers gRPC. Protobuf schemas define service contracts and message structures, which are compiled into language-specific data access classes. The binary wire format is far more compact than JSON, eliminating field name overhead and reducing serialization/deserialization CPU cost. For model serving, this means payloads are smaller and parsing is faster—critical when shaving milliseconds off prediction latency. Protobuf also enforces strong typing and backward compatibility through field numbering, allowing serving APIs to evolve without breaking existing clients.

The transport layer beneath gRPC that enables concurrent streams over a single TCP connection. Unlike HTTP/1.1's head-of-line blocking, HTTP/2 multiplexes many requests and responses simultaneously, eliminating connection setup overhead. Key features leveraged by gRPC:

• Binary framing: Efficient parsing of headers and data
• Stream prioritization: Clients can signal which inference requests are urgent
• Server push: Theoretically allows servers to preemptively send related predictions
• Flow control: Per-stream backpressure prevents any single client from saturating the server This is essential for high-throughput serving where thousands of clients maintain persistent connections.

gRPC natively supports four communication patterns beyond simple request-response, enabling sophisticated model serving architectures:

• Unary RPC: Single request, single response (standard classification/prediction)
• Server streaming: Client sends one request, receives a stream of tokens (ideal for LLM token generation via SSE-like behavior over gRPC)
• Client streaming: Client sends a stream of data (e.g., audio frames for real-time speech recognition)
• Bidirectional streaming: Both sides stream independently (useful for interactive dialogue systems where context accumulates) These patterns eliminate the need for custom WebSocket or polling logic.

gRPC provides built-in deadline propagation across service boundaries. A client sets an absolute deadline, and gRPC communicates this deadline to all downstream services in the call chain. If the deadline is exceeded, the request is automatically cancelled at every hop, freeing server resources immediately. For model serving, this prevents:

• Resource waste: No compute spent on predictions the client has already abandoned
• Cascading overload: Slow inference doesn't clog queues indefinitely
• Tail latency mitigation: Combined with circuit breakers, deadlines enforce SLOs at the protocol level Deadlines are distinct from simple timeouts—they represent an absolute point in time, not a duration.

gRPC's persistent HTTP/2 connections challenge traditional L4 load balancers, which distribute connections rather than requests. Effective gRPC load balancing requires:

• Proxy-based (L7): Envoy or Linkerd terminate HTTP/2 and distribute individual RPCs across backend model servers
• Client-side: The gRPC client library maintains a connection pool and uses algorithms like round-robin or least-loaded to select endpoints
• Look-aside: An external load balancer provides the client with a list of healthy server addresses, and the client handles distribution For model serving, client-side load balancing with health checking is often preferred, as it eliminates the proxy hop latency and enables direct streaming connections to GPU-backed instances.

gRPC interceptors are the equivalent of HTTP middleware, allowing cross-cutting concerns to be applied to every RPC without modifying service logic. Common serving use cases:

• Authentication: Inject JWT validation or mTLS certificate checks
• Logging and tracing: Capture request metadata, latency, and errors for OpenTelemetry export
• Rate limiting: Enforce per-client quotas before requests reach the model
• Retry logic: Automatically retry failed requests with exponential backoff
• Metrics collection: Record request counts, latency histograms, and error rates for SLO monitoring Interceptors operate at both client and server sides, forming a composable processing pipeline.