Custom Resource Definition (CRD)

A Custom Resource Definition (CRD) is a Kubernetes API extension that allows users to define new resource types (Kinds) and add them to the cluster's API server. Once registered, these custom resources can be created, read, updated, and deleted using standard Kubernetes tools like kubectl, just like native resources such as Pods or Deployments. This transforms Kubernetes from a generic container orchestrator into a domain-specific platform for managing any declarative state.

• Example: Defining a InferenceService CRD to represent an LLM endpoint.
• Result: Users can run kubectl get inferenceservices to list all deployed models.

A custom resource is a declarative YAML/JSON manifest that describes the desired state of a custom object. The spec field contains the user's configuration, while the status field is managed by a controller to reflect the actual state. This pattern enforces a clean separation between user intent and system reality.

• Structure: apiVersion, kind, metadata, spec, status.
• Example: An InferenceService spec might define the model ID, compute resources, and autoscaling rules.

A custom resource is inert without a controller (often implemented as an Operator). The controller runs a continuous reconciliation loop, watching for changes to custom resources. It compares the spec (desired state) with the status (observed state) and executes imperative logic—like deploying Pods or configuring services—to drive the cluster toward the desired state. This is the core automation mechanism for CRDs.

• Watch: The controller subscribes to API events for the custom resource.
• Diff & Act: It computes the difference between spec and status and takes corrective actions.
• Update Status: It writes the current observed state back to the resource's status field.

CRDs can include a structured validation schema using OpenAPI v3, which is enforced by the Kubernetes API server upon creation or update. This prevents invalid configurations from being stored. The schema defines required fields, data types, and value constraints for the spec.

• Benefit: Catches configuration errors immediately at the API layer.
• Example: Validating that a replicaCount field is an integer greater than zero.
• Tooling: Schemas enable better IDE support and documentation generation.

Custom resources integrate seamlessly with native Kubernetes ecosystems. They can be referenced by other resources, managed by tools like Helm, secured via RBAC, and monitored through standard APIs. This allows custom abstractions to leverage the full power of the platform.

• RBAC: Define roles that grant permissions to create or watch custom resources.
• Helm: Package and deploy a CRD and its controller as a single chart.
• Service Mesh: A custom SmartRouter resource could generate Istio VirtualService configuration.

CRDs are the essential data model for the Operator Pattern, a method of packaging and deploying complex, stateful applications on Kubernetes. An Operator consists of one or more CRDs (to define the application's domain) and a controller (to manage its lifecycle). This pattern is critical for LLMOps, enabling the declarative management of resources like InferenceService, LLMFineTuneJob, or PromptVersion.

• Use Case: An LLM Serving Operator uses an InferenceService CRD to manage model deployments, autoscaling, and canary rollouts.
• Automation: The operator handles complex operational tasks like GPU provisioning and model caching, reducing manual toil.

A Controller is a core Kubernetes concept and the active component of an Operator. It is a control loop that:

• Watches the state of specified resources (both built-in like Pods, and custom ones defined by CRDs).
• Compares the observed state with the desired state specified in the resource's spec.
• Makes Changes via the Kubernetes API to drive the observed state toward the desired state. This process is called reconciliation.

For a CRD like InferenceService, the custom controller is responsible for interpreting its spec and creating the necessary Deployments, Services, and ConfigMaps to realize a running inference endpoint.

A Custom Resource (CR) is an instance of a Custom Resource Definition (CRD). While the CRD defines the schema (the type of object, like InferenceService), the CR is a concrete instance of that type with specific values.

• Example: If a CRD defines an InferenceService with fields for modelName and replicas, a CR would be a YAML file declaring modelName: mixtral-8x7b and replicas: 2.
• Declarative API: The CR's spec field contains the user's desired state. The associated controller reads this spec and acts to make it true.
• Storage: CRs are persistent objects stored in the Kubernetes API server's backing store (etcd), just like native resources such as Pods or Deployments.

The Horizontal Pod Autoscaler (HPA) is a built-in Kubernetes controller that automatically scales the number of Pod replicas in a Deployment or StatefulSet based on observed metrics like CPU utilization or custom metrics.

• Integration with CRDs: A custom controller for an InferenceService CRD would typically create a standard Kubernetes Deployment. It could then also create an HPA resource targeting that Deployment to enable automatic scaling based on inference request load (e.g., queries per second).
• Declarative Scaling: This allows the LLM serving infrastructure to be defined declaratively: the CR specifies the initial replica count, and the HPA policy defines how it should scale under load, all managed by Kubernetes.