Question

You operate a 40-node GPU cluster and need continuous, fleet-wide visibility into per-GPU utilization, memory usage, temperature, and ECC error counts. Your platform team already runs Prometheus for metrics collection and Grafana for dashboards. Which approach best integrates NVIDIA GPU telemetry into this existing observability stack?

Accepted Answer

Deploy DCGM-Exporter on each node to expose GPU metrics in a Prometheus-scrapable format, then visualize them in Grafana

Answer

Run nvidia-smi in a cron job on each node and parse the text output into log files for manual review

Answer

Enable the GPU driver's built-in SNMP agent and point Prometheus at each node's SNMP endpoint

Answer

Rely on Kubernetes node-exporter, which automatically reports GPU utilization and ECC errors