Load Balancer

Load balancers use various algorithms to decide which backend server receives a request. Common strategies include:

• Round Robin: Distributes requests sequentially across all healthy servers.
• Least Connections: Routes traffic to the server with the fewest active connections, ideal for long-lived inference sessions.
• IP Hash: Uses the client's IP address to determine the target server, ensuring session affinity for stateful interactions.
• Weighted Distribution: Assigns requests based on server capacity (e.g., routing more traffic to GPU-rich nodes).

To ensure reliability, load balancers continuously probe backend servers. They send periodic health checks (e.g., HTTP GET requests to a /health endpoint) to verify a server is operational. If a server fails multiple consecutive checks, it is automatically drained from the pool. New requests are routed only to healthy servers, while existing connections may be gracefully terminated or allowed to complete. This process provides high availability by eliminating single points of failure in the inference cluster.

For certain model serving scenarios, it is necessary to route a user's subsequent requests to the same backend server. This is called session persistence or sticky sessions. The load balancer achieves this by injecting a cookie or using the client's IP address. This is critical when:

• The model server maintains an in-memory KV cache for a specific user session.
• The inference state is stored locally on a server (e.g., for a conversational agent). Without persistence, subsequent requests might hit a different server lacking the necessary context, causing errors or redundant computation.

Load balancers often handle the decryption of incoming HTTPS traffic, a process called SSL/TLS termination. This offloads the computationally expensive decryption/encryption work from the backend inference servers, allowing them to dedicate resources to model execution. The load balancer communicates with backend servers over an internal, unencrypted network (or re-encrypts for an additional security layer). This centralization also simplifies certificate management, as SSL certificates are installed and updated only on the load balancer.

In Kubernetes-based model serving, load balancing is a native construct. The Kubernetes Service (of type LoadBalancer or ClusterIP) provides internal load balancing across a set of identical Pods running an inference server. For external traffic, an Ingress controller (like NGINX Ingress or AWS ALB Ingress Controller) acts as a sophisticated HTTP(S) load balancer, providing routing, SSL termination, and name-based virtual hosting. Tools like KServe build upon these primitives to provide advanced, model-aware traffic management and canary deployments.

Modern load balancers and API gateways offer features for sophisticated traffic control:

• Rate Limiting: Enforces quotas on requests per client, API key, or model endpoint to prevent abuse and ensure fair resource sharing.
• Canary & Blue-Green Deployments: Enables gradual rollout of new model versions by routing a percentage of traffic (e.g., 5%) to the new version while monitoring for errors or performance regression.
• Request/Response Transformation: Can modify headers, paths, or payloads before they reach the inference server, aiding in versioning and integration.
• Latency-Based Routing: Directs requests to the backend server or geographic region with the lowest observed latency.

A reverse proxy that sits between clients and your services, acting as a single entry point. It handles cross-cutting concerns before traffic reaches the load balancer. Core functions:

• Authentication & Authorization: Validates API keys or tokens.
• Rate Limiting: Prevents any single client from overwhelming the system.
• Request Transformation: Modifies request/response formats (e.g., JSON to protobuf).
• Logging & Monitoring: Centralizes access logs and metrics collection.

A dedicated infrastructure layer for managing service-to-service communication in a microservices architecture, which includes model inference pods. It provides:

• Advanced Traffic Management: Fine-grained routing rules (e.g., send 10% of traffic to a new model version).
• Resilience Features: Automatic retries, timeouts, and circuit breaking for failed inference calls.
• Observability: Detailed telemetry (latency, errors) for all interservice calls.
• mTLS Security: Encrypts traffic between all pods, including those behind the load balancer.

The native Kubernetes abstraction for exposing a set of pods (e.g., inference server pods) as a network service. It is the fundamental layer a cloud load balancer often integrates with.

• ClusterIP: Internal service IP for load balancing within the cluster.
• NodePort & LoadBalancer: Exposes the service externally; cloud providers automatically provision a managed load balancer for type: LoadBalancer.
• Selectors: Uses labels to dynamically find and include healthy pods in the load-balanced pool.

The mechanism that automatically adjusts the number of active inference server instances (pods) based on demand. Works in tandem with the load balancer.

• Horizontal Pod Autoscaler (HPA): Scales the number of pods based on CPU/memory usage or custom metrics (e.g., requests per second).
• Cluster Autoscaler: Adds or removes worker nodes from the cluster itself.
• Dynamic Pool: As new pods scale up, the load balancer's target group is automatically updated to include them.

A critical function where the load balancer periodically probes backend inference servers to determine their availability.

• Liveness Probe: Determines if a pod is running. Failure results in the pod being restarted and removed from the load balancer pool.
• Readiness Probe: Determines if a pod is ready to accept traffic. Failure results in the pod being temporarily removed from the load balancer pool until it recovers.
• Configurable: Path (e.g., /health), port, interval, and success thresholds are defined in the deployment configuration.