GPU Operator

Manages the entire lifecycle of NVIDIA GPU drivers as Kubernetes DaemonSets. The operator detects the GPU hardware on each node and automatically deploys the correct, validated driver version.

• Eliminates manual driver installation on bare-metal hosts
• Ensures driver consistency across the entire cluster
• Supports pre-compiled, pre-validated driver containers for air-gapped environments

Deploys the NVIDIA Container Toolkit and configures the container runtime (containerd or CRI-O) to expose GPUs to containers. This includes the nvidia-container-runtime hook and the nvidia-container-toolkit binary.

• Configures the OCI runtime spec to mount GPU device nodes
• Enables the nvidia.com/gpu resource type in Kubernetes
• Manages the libnvidia-container library stack for GPU enumeration

Deploys the NVIDIA Data Center GPU Manager (DCGM) and the DCGM Exporter to expose GPU metrics to Prometheus. Provides deep hardware telemetry for operational visibility.

• Exposes metrics: GPU utilization, memory usage, temperature, power draw, ECC errors
• Integrates with the Prometheus Operator via ServiceMonitor CRDs
• Enables GPU-specific Horizontal Pod Autoscaler (HPA) rules based on DCGM metrics

Uses the GPU Feature Discovery (GFD) component to automatically label nodes with their GPU properties. This enables intelligent scheduling based on hardware capabilities.

• Labels: nvidia.com/gpu.product, nvidia.com/gpu.count, nvidia.com/gpu.memory
• Enables node affinity and node selector rules for specific GPU models
• Supports NVIDIA MIG (Multi-Instance GPU) partitioning discovery and labeling

Manages its own lifecycle through a ClusterPolicy Custom Resource Definition (CRD). The ClusterPolicy is the single declarative configuration point for the entire GPU software stack.

• Validates component versions and driver-container compatibility
• Supports air-gapped deployments via private container registries
• Integrates with the Node Feature Discovery (NFD) operator for hardware enumeration

Automates the configuration of Multi-Instance GPU (MIG) partitioning on supported NVIDIA data center GPUs (A100, H100). The operator can declaratively define MIG geometry.

• Defines MIG profiles via the ClusterPolicy CRD
• Creates isolated GPU instances with dedicated memory and compute slices
• Enables fine-grained GPU sharing for inference workloads without virtualization overhead

A scheduling mechanism that dedicates GPU-equipped nodes to specific AI workloads. The GPU Operator automatically applies taints to nodes with specialized hardware, ensuring only pods with matching tolerations are scheduled.

• NoSchedule taint: Prevents non-GPU pods from landing on expensive hardware
• nvidia.com/gpu toleration: Required for any pod requesting GPU resources
• Prevents resource contention between training and inference jobs

A hardware-level partitioning technology that slices a single physical GPU into up to 7 isolated instances, each with dedicated compute, memory, and cache. The GPU Operator manages MIG configuration declaratively.

• Enables right-sizing GPU allocation for inference vs. training
• Each MIG instance appears as a separate Kubernetes resource
• Critical for maximizing utilization in air-gapped clusters with limited hardware

A technology stack enabling direct data transfer between GPU memory and network interface cards, bypassing system memory and CPU. The GPU Operator installs the required NVIDIA Peer Memory drivers and configures the nvidia-fs components.

• Reduces latency for distributed training across multiple nodes
• Essential for NCCL collective communication in air-gapped clusters
• Requires compatible NICs and SR-IOV configuration

The GPU Operator deploys DCGM Exporter (Data Center GPU Manager) to expose hardware telemetry as Prometheus metrics. This is the primary observability pipeline for GPU health in disconnected clusters.

• DCGM_FI_DEV_GPU_UTIL: GPU utilization percentage
• DCGM_FI_DEV_FB_USED: Framebuffer memory consumption
• DCGM_FI_DEV_ECC_ERRORS: Error-correcting code failure counts
• Enables HPA scaling based on custom GPU metrics