Connection Pooling

Establishing a new TCP connection requires a three-way handshake, and TLS adds multiple round trips for cryptographic negotiation. For a typical inference request that completes in 10ms, a 50ms handshake represents a 500% overhead. Connection pooling amortizes this cost by keeping connections alive, reducing the per-request setup time to effectively zero after the initial warm-up. This is especially critical in microservice architectures where a single user request may fan out to dozens of downstream model endpoints.

A connection pool actively governs the lifecycle of each persistent socket. Key mechanisms include:

• Keep-alive probes: Periodic TCP keep-alive packets detect silently broken connections caused by intermediate network devices with idle timeouts.
• Connection eviction: Stale or idle connections exceeding a configurable Time-to-Live (TTL) are proactively closed to prevent resource leaks.
• Health checks: Pre-request validation ensures a borrowed connection is still viable before handing it to a client thread, avoiding failed inference calls.
• Exponential backoff: When a backend is unreachable, the pool backs off reconnection attempts to avoid thundering herd problems during recovery.

An unbounded number of connections can overwhelm both the client and server, exhausting file descriptors and ephemeral ports. A properly configured pool enforces hard limits:

• Max total connections: The absolute ceiling on simultaneous open sockets.
• Max per-route connections: Limits connections to a specific backend host to prevent a single service from monopolizing resources.
• Connection timeout: The maximum time a client thread will block waiting for an available connection before throwing an exception, triggering a circuit breaker or fallback logic. These bounds are essential for maintaining predictable P99 latency under load.

While HTTP/1.1 connection pools manage one request per connection at a time, HTTP/2 and gRPC introduce multiplexed streams over a single TCP connection. In this paradigm, the pool manages a smaller number of long-lived connections, each carrying hundreds of concurrent streams. This dramatically reduces the connection count but shifts complexity to stream-level flow control. A single stalled stream can block the entire connection if not properly managed, making HTTP/2-specific pool tuning—like max concurrent streams and frame size limits—critical for inference workloads using gRPC streaming.

Sizing a connection pool for model serving requires balancing latency and resource consumption. A common heuristic is to set the pool size to the expected peak concurrent requests. However, for GPU-backed inference, the bottleneck is often the model's batch processing capacity, not the connection count. Over-provisioning connections simply creates deeper queues at the model server, increasing queuing latency. The optimal pool size should be derived from Little's Law: L = λ * W, where L is the number of connections needed, λ is the request arrival rate, and W is the average request latency. This ensures the pool is large enough to sustain throughput without wasteful idle connections.

When all connections in a pool are in use and the wait queue is full, the system enters a state of connection pool exhaustion. Without defensive patterns, this cascades into thread starvation and request timeouts. Mitigations include:

• Circuit Breaker: Trips when a backend's error rate or latency exceeds a threshold, immediately failing requests to prevent resource drain.
• Bulkheading: Isolates connection pools per downstream service so a failure in one model endpoint does not exhaust connections needed by another.
• Graceful degradation: Returns a cached or default prediction when the primary model pool is saturated, maintaining user experience under stress.