GPU Operator

The operator manages the complete lifecycle of NVIDIA GPU drivers as Kubernetes DaemonSets, ensuring every GPU node runs a compatible, validated driver version. It handles initial deployment, rolling updates, and node-specific configurations without manual SSH intervention.

• Precompiled Driver Containers: Ships drivers as OCI-compliant container images, eliminating dependency on the host OS package manager.
• Node Selector Targeting: Automatically deploys drivers only to nodes with NVIDIA hardware, detected via PCI device labels.
• Rolling Update Strategy: Orchestrates node cordoning and draining to update drivers without disrupting running GPU workloads.

The operator configures the NVIDIA Container Toolkit and the container runtime (e.g., containerd, CRI-O) to expose GPUs to containers. It injects the nvidia-container-runtime as a prestart hook, enabling device node and library mounting.

• RuntimeClass Registration: Creates a nvidia RuntimeClass in Kubernetes, allowing pods to request GPU resources via runtimeClassName: nvidia.
• CDI (Container Device Interface): Modern versions use CDI to declaratively define which device nodes, libraries, and binaries are injected into containers.
• Multi-Runtime Support: Simultaneously configures multiple runtimes (Docker, containerd, CRI-O) on the same node for migration scenarios.

The operator deploys GPU Feature Discovery (GFD) to automatically detect and advertise GPU properties as Kubernetes node labels. This enables intelligent scheduling based on hardware capabilities without manual node annotation.

• Labels Discovered: nvidia.com/gpu.product, nvidia.com/gpu.memory, nvidia.com/gpu.family, nvidia.com/cuda.driver-version.
• MIG Discovery: Detects Multi-Instance GPU partitioning configurations and exposes available MIG slices as schedulable resources.
• Dynamic Updates: Labels are refreshed on a configurable interval to reflect hardware changes or driver updates.

The operator deploys NVIDIA DCGM Exporter to expose GPU metrics in Prometheus format, integrating with the cluster's existing observability stack. This provides real-time visibility into GPU utilization, temperature, power draw, and error conditions.

• Metrics Exposed: GPU utilization, memory usage, temperature, power consumption, ECC errors, PCIe throughput, and NVLink bandwidth.
• XID Error Detection: Captures and exposes NVIDIA XID errors that indicate hardware faults, enabling automated alerting.
• Grafana Dashboards: Ships with pre-built dashboards for visualizing GPU health across the entire cluster.

The operator automates the configuration of Multi-Instance GPU (MIG) partitioning on supported NVIDIA data center GPUs (A100, H100, B200). It applies declarative MIG configurations, carving a single physical GPU into isolated, right-sized instances.

• Declarative MIG Profiles: Define MIG geometry via Kubernetes custom resources, specifying slice counts and sizes (e.g., 1g.5gb, 2g.10gb).
• Reboot Persistence: MIG configurations survive node reboots, applied automatically by the operator on node startup.
• Dynamic Reconfiguration: Supports changing MIG partitions on idle GPUs without manual nvidia-smi commands.

The operator includes a GPU Operator Validator component that runs a suite of smoke tests to verify the entire GPU software stack is functioning correctly after deployment. This catches misconfigurations before production workloads are scheduled.

• CUDA Validation: Runs a simple CUDA kernel to confirm the compiler, driver, and device are communicating.
• Peer-to-Peer Test: Validates NVLink and GPUDirect P2P connectivity between GPUs on the same node.
• RDMA Verification: Tests GPUDirect RDMA over InfiniBand or RoCE fabrics for multi-node communication readiness.