Custom Resource Definition (CRD)

A CRD defines a new RESTful API endpoint in the Kubernetes API server by specifying an OpenAPI v3 schema. This schema enforces strict validation on the structure, data types, and required fields of custom resources.

• Structural Schemas: Define the exact JSON structure, preventing malformed configuration from being persisted to etcd.
• Pruning: Unknown fields not defined in the schema are automatically stripped, ensuring a clean, predictable spec.
• Defaulting: Schemas can specify default values for fields, simplifying the user experience for complex AI workload configurations.

A CRD without a controller is just a data store. The true power lies in the operator pattern, where a custom controller continuously watches for changes to custom resources and executes a reconciliation loop to drive the actual system state toward the desired state declared in the spec.

• Level-Triggered Logic: Controllers react to the current state, not individual events, making them resilient to missed updates.
• Idempotency: Reconciliation logic must be idempotent; running it multiple times produces the same result, ensuring stability.
• Example: An NVIDIA GPU Operator uses a CRD to manage the lifecycle of GPU drivers and device plugins, reconciling the desired driver version with what is installed on each node.

CRDs support multiple API versions simultaneously, enabling the evolution of an API without breaking existing clients. Conversion webhooks handle the translation of objects between versions on the fly.

• Served vs. Storage Versions: A CRD can serve multiple versions (e.g., v1beta1 and v1) but stores only one version in etcd. The conversion webhook translates between the served and storage versions.
• Round-Tripping: Conversion must be lossless; converting from v1 to v1beta1 and back to v1 must yield the identical object.
• Deprecation: Individual versions can be deprecated with a warning message, guiding users toward the stable API.

CRDs can expose standard Kubernetes subresources like /status and /scale to separate the specification of desired state from the reporting of observed state, a critical pattern for AI workloads.

• Status Subresource: The controller writes observed state (e.g., 'TrainingJob is Running') to the /status endpoint, while the user writes desired state to the /spec. This prevents users from accidentally overwriting controller-managed fields.
• Scale Subresource: Enables native integration with the Horizontal Pod Autoscaler (HPA). An AI inference CRD can expose a /scale endpoint, allowing HPA to automatically adjust the number of inference server replicas based on custom metrics like request latency.

CRDs integrate seamlessly with Kubernetes admission webhooks to implement complex, stateful validation and mutation logic that goes beyond static OpenAPI schemas.

• Validating Webhooks: Reject the creation or update of a custom resource if it violates business logic, such as ensuring an AI training job's requested GPU count does not exceed a namespace quota.
• Mutating Webhooks: Automatically inject default configuration into a custom resource, like adding a sidecar container for a monitoring agent or injecting a pre-configured volume mount for a shared dataset.
• CEL Validation: For simpler rules, Common Expression Language (CEL) can be embedded directly in the CRD schema for lightweight, in-process validation without a webhook call.

CRDs can define additional printer columns to display key information when a user runs kubectl get <resource>. This transforms opaque custom resources into observable, operator-friendly entities.

• JSONPath Columns: Map a specific field from the resource's JSON structure to a column, such as displaying the status of an AI training job (Succeeded, Failed, Running).
• Priority and Format: Columns have a priority level (0 for standard view, 1 for wide view with -o wide), and the format can be specified (e.g., date, integer, boolean).
• Example: A kubectl get trainingjobs command could instantly show the Job Name, Status, Age, and GPU Hours Consumed, providing immediate situational awareness to a platform engineer.