GPU RAS

Modern GPUs implement Error Correction Code (ECC) memory to detect and correct single-bit errors and detect double-bit errors in HBM3e and on-chip SRAM. This protects against data corruption caused by cosmic radiation or electrical noise. Key mechanisms include:

• SEC-DED: Single Error Correction, Double Error Detection for memory arrays
• SECDED on register files and caches to protect computational state
• Link-level CRC and retry on NVLink and PCIe interfaces to guarantee data integrity in transit
• End-to-end data poisoning to prevent corrupted results from propagating through the system

GPUs expose a rich set of out-of-band telemetry through the Redfish API and in-band via DCGM, enabling proactive fault prediction. Monitored parameters include:

• Junction and memory temperatures with configurable throttling thresholds
• Power draw in milliwatts across multiple rails for anomaly detection
• Clock frequencies and utilization to identify degraded performance states
• Fan speeds and coolant flow rates for direct liquid cooling systems This data feeds into predictive maintenance models that can forecast component failures before they cause workload interruptions.

Before production deployment, GPUs undergo rigorous burn-in testing to identify infant mortality failures. Diagnostic suites include:

• CUDA memtest for exhaustive memory pattern testing under thermal load
• NVSwitch and NVLink loopback tests to validate interconnect integrity
• Computational stress tests running dense matrix operations at maximum TDP
• DCGM diagnostic levels that progressively escalate from quick health checks to deep, disruptive validation These tests establish a hardware reliability baseline and are often integrated into automated bare-metal provisioning pipelines.

Beyond memory page retirement, GPUs incorporate hardware-level redundancy to survive component failures:

• Spare streaming multiprocessors that can be transparently activated if a primary SM fails POST
• Redundant NVSwitch paths that maintain multi-GPU connectivity even with a failed switch port
• Power phase redundancy in voltage regulator modules to tolerate individual phase failures
• Fuse-based disable mechanisms that permanently isolate defective units identified during manufacturing test These features enable graceful degradation rather than catastrophic failure, critical for large-scale training jobs running across thousands of GPUs.

The Data Center GPU Manager is NVIDIA's essential telemetry and health-monitoring toolkit. It provides real-time access to GPU sensors, including XID errors, ECC error counts, PCIe replay counters, and thermal throttling events. DCGM is the primary data source for building automated RAS alerting and predictive failure analysis pipelines in large-scale clusters.

A rigorous pre-production validation process that subjects new GPUs to sustained maximum thermal and computational load. The goal is to force infant mortality failures to surface before hardware enters production. Burn-in tests typically run for 24-72 hours, executing dense matrix operations while monitoring for ECC errors, memory faults, and thermal instability to ensure only reliable hardware is deployed.

High Bandwidth Memory 3e is the latest stacked DRAM standard powering modern AI accelerators. Its vertical die stacking provides over 1 TB/s of memory bandwidth per GPU, critical for large model training. From a RAS perspective, HBM3e integrates advanced on-die ECC and error scrubbing mechanisms to detect and correct single-bit errors in the vast memory arrays, directly preventing silent data corruption in model weights.

A thermal management method that circulates a dielectric coolant directly to heat-generating components like GPUs. By efficiently removing heat at the source, DLC prevents thermal throttling, a primary cause of degraded GPU availability and performance. Maintaining stable, lower junction temperatures directly increases the Mean Time Between Failures (MTBF) of high-power accelerators.