Load Balancer

Load balancers use specific algorithms to decide which backend server receives a client request. Common algorithms include:

• Round Robin: Distributes requests sequentially across the server pool.
• Least Connections: Routes traffic to the server with the fewest active connections.
• IP Hash: Uses the client's IP address to determine the server, ensuring a user consistently reaches the same backend (session persistence).
• Weighted Round Robin/Least Connections: Assigns a weight to each server based on capacity (CPU, RAM), directing more traffic to higher-capacity nodes. The choice of algorithm directly impacts load distribution efficiency and is critical for applications requiring sticky sessions.

Load balancers continuously monitor the health of backend servers using health checks (e.g., HTTP GET requests, TCP pings). If a server fails a health check, the load balancer automatically stops sending traffic to it, performing a failover to healthy instances. This is fundamental for high availability (HA). Common probe types include:

• Liveness Probe: Determines if the server process is running.
• Readiness Probe: Determines if the server is ready to accept traffic (e.g., warmed up, connected to a database). This feature prevents user requests from being sent to failed or degraded servers, ensuring application resilience.

Also known as session affinity, this feature ensures that all requests from a single user session are directed to the same backend server. This is essential for stateful applications where user session data is stored locally on a server (e.g., in-memory sessions, local caches). The load balancer typically uses a cookie or the client's IP address to maintain this mapping. Without session persistence, users could lose their application state if subsequent requests land on a different server. It's a trade-off between perfect load distribution and user experience for stateful services.

The load balancer can handle the decryption of incoming SSL/TLS-encrypted traffic (HTTPS) and pass unencrypted HTTP requests to the backend servers. This process, called SSL Offloading, provides significant benefits:

• Reduces computational load on backend servers, freeing CPU cycles for application logic.
• Centralizes certificate management on the load balancer.
• Simplifies backend server configuration. For enhanced security, some architectures use SSL Passthrough, where the load balancer forwards encrypted traffic without decrypting it, leaving end-to-end encryption intact.

Load balancers can enforce policies to control the flow of traffic, protecting backend services from being overwhelmed. Key capabilities include:

• Rate Limiting: Restricts the number of requests a client or IP can make in a given time window (e.g., 1000 requests per minute).
• Connection Throttling: Limits the number of concurrent connections from a single source.
• Quality of Service (QoS): Prioritizes certain types of traffic (e.g., API calls from a premium partner) over others. These features are crucial for DDoS mitigation, ensuring fair usage, and maintaining service stability during traffic spikes.

Modern cloud load balancers integrate seamlessly with auto-scaling groups. When an auto-scaling policy triggers (e.g., due to high CPU utilization), new server instances are launched automatically. The load balancer's health check system detects these new instances and seamlessly registers them into the backend pool, beginning to distribute traffic to them. Conversely, when scale-down occurs, instances are gracefully drained (stop receiving new connections) and then deregistered. This creates a fully elastic, self-healing infrastructure that optimizes for both performance and cost.

The primary function is to act as a reverse proxy, accepting client requests and distributing them across a pool of backend model servers or inference endpoints. This prevents any single server from becoming a bottleneck. Key distribution algorithms include:

• Round Robin: Distributes requests sequentially to each server in the pool.
• Least Connections: Routes traffic to the server with the fewest active connections.
• IP Hash: Uses the client's IP address to determine the target server, ensuring session persistence. For LLMs, distribution must account for variable request complexity, as a single long-context query can monopolize a GPU for seconds.

LLM inference presents unique load characteristics that generic load balancers may not handle optimally:

• Variable Latency: Request completion time depends heavily on output token count and model parameters, unlike uniform HTTP requests.
• Stateful Sessions: Applications using long-running conversations or streaming responses require session affinity (sticky sessions) to route follow-up requests to the same backend instance holding the KV cache.
• GPU Memory Pressure: An overloaded model instance can exhaust VRAM, causing out-of-memory errors for subsequent requests, requiring health checks that monitor GPU status, not just HTTP liveness.

Load balancers continuously verify backend health using probes. For LLM servers, standard HTTP 200 OK may be insufficient.

• Liveness Probe: Confirms the inference server process is running (e.g., /health endpoint).
• Readiness Probe: Confirms the server is ready for inference, which requires checking if the model is loaded into GPU memory and the batch scheduler has capacity.
• Model-Specific Endpoints: In multi-model deployments, the balancer must discover which backends host specific model variants (e.g., Llama-3-70B vs. Mixtral-8x7B), often integrating with service registries or Kubernetes Custom Resource Definitions (CRDs).

In cloud-native LLM deployments, the load balancer is typically integrated with orchestration platforms:

• Kubernetes Service: A Service object with type LoadBalancer or Ingress controller automatically distributes traffic to pods running model servers.
• Horizontal Pod Autoscaler (HPA): The load balancer works in tandem with the HPA, which scales the number of backend pods based on metrics like average request latency or GPU utilization.
• Service Mesh: Tools like Istio or Linkerd provide advanced load balancing, traffic splitting for canary deployments of new model versions, and fine-grained observability into inter-service calls.

Beyond simple distribution, modern load balancers enable sophisticated deployment strategies critical for LLM ops:

• Traffic Splitting: Route a percentage of requests (e.g., 5%) to a new model version for A/B testing or canary analysis, monitoring for regressions in latency or output quality.
• Rate Limiting & Quotas: Enforce request limits per API key or user to prevent abuse of expensive inference resources and ensure fair usage.
• Priority Queuing: Implement queuing policies to prioritize low-latency interactive chat requests over high-latency batch processing jobs, preventing head-of-line blocking.

A dedicated infrastructure layer for managing service-to-service communication within a microservices architecture. It provides fine-grained traffic management, security (mTLS), and observability features that complement a traditional load balancer.

• Key Components: Data plane (sidecar proxies like Envoy) and a control plane (e.g., Istio, Linkerd).
• Function: Handles internal traffic routing, retries, timeouts, and circuit breaking between services, while an API Gateway or load balancer typically manages north-south (external) traffic.

A design pattern used in distributed systems to prevent cascading failures. It detects failures and stops an application from repeatedly trying to execute an operation that is likely to fail.

• States: Closed (normal operation), Open (requests fail fast), Half-Open (testing for recovery).
• Integration: Often implemented within a service mesh or application code to work in tandem with a load balancer. If a backend instance fails health checks, the load balancer stops sending traffic, while the circuit breaker pattern prevents clients from waiting on timeouts.

A periodic test performed by an orchestrator or load balancer to verify that an application instance is running correctly and ready to accept traffic.

• Types: Liveness probes determine if a container is running (restarts if failed). Readiness probes determine if a container is ready to serve requests (removed from load balancer pool if failed).
• Purpose: Enables automatic failure detection. A load balancer uses these results to dynamically update its pool of healthy backend targets, ensuring traffic is only routed to operational instances.

The practice of controlling the volume and rate of network traffic sent to a service. It manages load, prevents overload, and ensures fair resource allocation.

• Techniques: Includes rate limiting (controlling request frequency per client) and prioritization queues.
• Relation to Load Balancing: While a load balancer distributes traffic, traffic shaping regulates it. They are often used together: a load balancer spreads requests across servers, while a shaper or rate limiter at the ingress point protects those servers from being overwhelmed by excessive traffic bursts.

A distributed hashing algorithm that minimizes reorganization when the number of nodes in a system changes. It is critical for stateful load balancing and distributed caches.

• Problem it Solves: Traditional hashing (e.g., hash(key) % N) requires remapping most keys when N (number of servers) changes, causing cache misses and session disruption.
• Load Balancer Use: Used in load balancers (like HAProxy) for sticky sessions and in systems like Cassandra for data partitioning. It ensures a user's requests are consistently routed to the same backend server with minimal disruption during scaling events.

A cloud computing capability that automatically adjusts the number of active compute resources (e.g., servers, containers) based on real-time demand.

• Triggers: Metrics like CPU utilization, request queue length, or custom application metrics.
• Symbiosis with Load Balancing: Auto-scaling groups add or remove backend instances. The load balancer continuously discovers these instances via integration with the cloud provider's API (e.g., AWS Target Groups, GCP Instance Groups) and automatically begins or stops routing traffic to them. This creates a fully elastic, self-healing system.