DCGM

Actively tracks a wide array of GPU telemetry, including temperature, power draw, clock speeds, and memory utilization. It goes beyond basic metrics to monitor critical error states like XID errors, ECC error counts, and PCIe link integrity. This allows administrators to identify degraded hardware before it causes application failure.

• Monitors GPU, memory, and thermal throttling events
• Tracks volatile and non-volatile error counters
• Provides a holistic view of physical and logical device health

Includes built-in diagnostic tools to run active tests on GPU hardware. The dcgmi diag command executes a hierarchy of tests, from quick deployment checks to exhaustive stress tests, verifying compute integrity, memory bandwidth, and high-speed interconnect functionality. This is critical for burn-in testing and pre-production validation.

• Runs non-destructive, configuration-level checks (Level 1)
• Performs detailed parametric and stress tests (Levels 2-4)
• Validates NVLink and InfiniBand connectivity for multi-GPU setups

Enables the creation of custom health policies that trigger automated actions when specific error thresholds are met. For example, a policy can automatically mark a GPU as 'unhealthy' and migrate workloads away if a certain number of XID errors occur within a time window. This forms the basis for self-healing cluster orchestration.

• Defines rules based on any monitored metric or error code
• Integrates with workload schedulers like Slurm and Kubernetes
• Prevents faulty nodes from corrupting distributed training jobs

Provides detailed performance metrics with minimal impact on running workloads. It exposes key utilization data like SM occupancy, Tensor Core activity, and FP16/FP32 engine usage. This data is essential for identifying compute inefficiencies and optimizing CUDA kernel execution without intrusive profiling tools.

• Exposes fine-grained engine utilization metrics
• Tracks NVLink and PCIe bandwidth utilization
• Enables performance bottleneck identification in real-time

Designed as a foundational telemetry layer for the broader data center ecosystem. It exports metrics in standard formats like Prometheus for visualization in Grafana dashboards and integrates directly with the NVIDIA GPU Operator for automated deployment in Kubernetes clusters. This ensures GPU health data is a first-class citizen in any observability stack.

• Native Prometheus endpoint for metric scraping
• Core component of the GPU Operator for automated lifecycle management
• Powers higher-level tools like NVIDIA Triton Inference Server metrics

Can be configured to record key performance and health metrics to a time-series database. This historical data is invaluable for post-mortem analysis of job failures, identifying long-term degradation trends, and capacity planning. It provides the forensic evidence needed to distinguish between hardware faults and software bugs.

• Records metrics for long-term trend analysis
• Enables root cause analysis for intermittent failures
• Supports data-driven hardware refresh and procurement decisions