Question

During routine fleet monitoring, DCGM reports a steadily increasing count of correctable (single-bit) ECC memory errors on one GPU in a training node. The GPU continues to complete jobs successfully, but the error count grows week over week. What is the most appropriate operations response?

Accepted Answer

Track the trend and allow row remapping/page retirement to handle affected memory locations, drain and investigate only if uncorrectable errors appear or retired pages exceed vendor thresholds

Answer

Immediately RMA the GPU because any correctable ECC error indicates imminent hardware failure and data corruption

Answer

Disable ECC on the GPU to stop the error counter from incrementing and improve available memory bandwidth

Answer

Ignore the metric entirely since correctable errors are automatically fixed and never affect reliability