Kubernetes Operator

A Custom Resource Definition (CRD) is the Kubernetes API extension mechanism that allows users to define their own resource types. It is the foundational object that an Operator is built to manage.

• Core Concept: A CRD defines the schema (the spec and status fields) for a new custom object, like InferenceService or LLMDeployment.
• Operator's Domain: The Operator's controller watches for events (create, update, delete) on instances of this custom resource.
• Example: A CRD for an LLM might have fields for modelId, replicaCount, quantization, and autoscaling.minReplicas.

The Horizontal Pod Autoscaler (HPA) is a native Kubernetes controller that automatically scales the number of Pods in a deployment based on observed CPU, memory, or custom metrics. Operators often integrate with or manage HPAs.

• Automated Scaling: For LLM serving, an Operator might create an HPA that scales inference pods based on GPU memory utilization or request queue length.
• Operator Synergy: While HPA handles reactive scaling, the Operator manages the higher-level application lifecycle, ensuring new pods are correctly configured with the model and dependencies.

A Service Mesh is a dedicated infrastructure layer for managing service-to-service communication in a microservices architecture. When Operators deploy complex LLM applications, a service mesh handles cross-cutting concerns.

• Traffic Management: Enables fine-grained traffic routing for canary deployments of new model versions or A/B testing.
• Observability & Security: Provides automatic metrics, logs, traces (distributed tracing), and mutual TLS for secure communication between model serving pods and other services.
• Example Tools: Istio and Linkerd are common service meshes used in Kubernetes environments.

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure through declarative configuration files. Kubernetes Operators are a realization of IaC for application management.

• Declarative State: Users declare the desired state of their LLM application in a YAML file (the Custom Resource). The Operator's reconciliation loop acts as the "IaC engine" to make the real cluster match that state.
• GitOps: Operators are a key enabler of GitOps, where the custom resource manifests stored in Git are the single source of truth for the application's configuration and deployment.

A Model Registry is a centralized repository for storing, versioning, and managing metadata for trained machine learning models. Operators interact with model registries to deploy specific model versions.

• Deployment Source: An Operator's custom resource (e.g., spec.modelUri) typically points to a model artifact in a registry like MLflow, Hugging Face Hub, or a private S3 bucket.
• Lifecycle Integration: The Operator pulls the specified model version, potentially applying quantization or other optimizations, and loads it into the inference engine (e.g., vLLM, Triton Inference Server).

Canary deployment is a release strategy where a new version of an application is deployed to a small subset of traffic initially. Operators can orchestrate sophisticated canary releases for LLM model updates.

• Risk Mitigation: Allows monitoring of a new model version's performance, latency, and accuracy before a full rollout.
• Operator Implementation: The Operator manages multiple deployments (stable and canary) and configures a Service Mesh or ingress controller to split traffic between them based on the custom resource's configuration.