AI Operations

Before dispatching a large multi-node training job on a newly repaired GPU node, an operator wants to run a comprehensive validation that stresses memory, executes compute tests, and checks integration health beyond a quick sanity check. Using the DCGM diagnostic tool (dcgmi diag), which run level should the operator select to perform this deeper, longer-running validation?

During routine fleet monitoring, DCGM reports a steadily increasing count of correctable (single-bit) ECC memory errors on one GPU in a training node. The GPU continues to complete jobs successfully, but the error count grows week over week. What is the most appropriate operations response?

You operate a 40-node GPU cluster and need continuous, fleet-wide visibility into per-GPU utilization, memory usage, temperature, and ECC error counts. Your platform team already runs Prometheus for metrics collection and Grafana for dashboards. Which approach best integrates NVIDIA GPU telemetry into this existing observability stack?

After a routine driver upgrade on a DGX system with NVSwitch, several multi-GPU training jobs fail immediately at startup with NCCL errors, and nvidia-smi shows all GPUs present and healthy. Single-GPU jobs run fine. Checking the host, you find the fabric manager service is not running. What is the most likely cause and correct remediation?

A cluster operator notices that several Slurm training jobs intermittently fail minutes after starting, always landing on the same few GPU nodes. The failures do not appear during normal idle periods. The operator wants to automatically prevent jobs from being scheduled onto GPUs that have degraded or failing hardware before the job begins. Which approach best addresses this?

A multi-GPU training job on a DGX node suddenly shows much lower throughput than a previous identical run on the same node. GPU utilization (SM activity) and power draw both read near 100%, temperatures are normal, and no XID errors appear in the logs. An administrator wants to confirm whether inter-GPU communication has degraded before escalating a hardware ticket. Which DCGM metric should the administrator examine first?

You administer a multi-node training cluster. Users report that a specific 8-GPU node occasionally produces corrupted gradient reductions during NCCL all-reduce operations, but only under heavy load. GPU utilization, power draw, and temperature all appear normal in your dashboards. You want to use DCGM to pinpoint whether an intermittent interconnect fault is the root cause. Which DCGM field is most appropriate to investigate first?

A cluster administrator receives complaints that GPU nodes appear busy but training throughput is low. They want to attribute measured GPU SM activity, memory usage, and energy to individual Slurm jobs so they can identify which jobs underutilize their allocated GPUs. Which approach provides per-job GPU accounting metrics with the least custom development?

An operations team wants to track energy efficiency across a large training cluster. They need a per-GPU telemetry field that reports real-time board power draw in watts so they can correlate power consumption with job phases and identify GPUs that are drawing significantly more power than peers under similar load. Which DCGM field group best serves this requirement?

You operate a 32-node GPU training cluster. Users report that jobs landing on one specific node run roughly 20% slower than identical jobs on other nodes, but no jobs are failing and no XID errors appear in the logs. You want to confirm whether the GPUs on that node are silently reducing performance before you take any disruptive action. Which DCGM field group should you examine first?

During a large multi-node training run, one node repeatedly crashes its training process. The cluster operator wants to determine whether a specific GPU is experiencing hardware faults such as ECC errors or falling off the bus. Which NVIDIA tool and data source should the operator use to programmatically collect GPU health events, XID error codes, and run active diagnostics before deciding whether to drain the node?

You administer a Kubernetes cluster with 12 GPU nodes running long-lived training jobs. You need to apply a new GPU driver version to one node, but you want the node to stop accepting NEW pods immediately while allowing the currently running training pods to finish naturally over the next several hours before you reboot. Which action best achieves this?

In a shared Kubernetes cluster, an operations team runs both latency-sensitive inference pods and long-running training jobs on GPU nodes. They want to reserve a dedicated pool of eight H100 nodes exclusively for the inference workload so that training jobs never land there, while inference pods are still guaranteed to schedule onto those nodes. Which combination of Kubernetes mechanisms best accomplishes this?

A platform engineer manages a Kubernetes cluster using the NVIDIA GPU Operator. After a rolling worker-node OS upgrade, GPU-requesting pods on the upgraded nodes stay in 'Pending' state, and `kubectl describe node` shows the nodes no longer advertise the `nvidia.com/gpu` extended resource. The nodes themselves are Ready, and other (non-GPU) pods schedule normally. What is the most likely root cause to investigate first?

You are deploying the NVIDIA GPU Operator on a Kubernetes cluster that contains a mix of nodes: some with A100 GPUs, some with L40S GPUs, and some CPU-only nodes. After installation, you notice GPU workloads are occasionally being scheduled to CPU-only nodes where they fail to start. Which component of the GPU Operator stack is responsible for automatically labeling nodes with their GPU characteristics so the scheduler can place GPU pods only on GPU-equipped nodes?

A Kubernetes cluster has four nodes, each with one NVIDIA GPU. A team submits eight lightweight inference pods, each requesting 'nvidia.com/gpu: 1'. Four pods schedule and run, but the other four remain in 'Pending' state indefinitely even though the running pods use less than 20% of GPU compute and memory. Cluster admins want all eight pods to run concurrently without buying more hardware. What is the most appropriate action?

On a shared Kubernetes GPU cluster, a high-priority production inference deployment cannot schedule because all GPUs are consumed by lower-priority batch training jobs that have no completion deadline. The platform team wants production pods to automatically evict batch pods when GPUs are scarce, without manually deleting workloads. Which Kubernetes mechanism should they configure to achieve this?

You administer a Kubernetes cluster serving a development team that runs many small, lightweight inference notebooks. Each notebook uses only a fraction of an A100 GPU's compute and memory, but developers complain that pods stay 'Pending' because each requests one whole GPU. You want to let multiple pods share a single physical GPU while requiring minimal reconfiguration and no guarantee of hardware-level isolation between workloads. Which approach best fits these requirements?

You administer a Kubernetes cluster with the NVIDIA GPU Operator. Some A100 nodes are configured with MIG enabled to serve many small inference pods, while other A100 nodes remain in full-GPU (non-MIG) mode for large training jobs. Data scientists report that inference pods requesting a MIG slice occasionally land on the full-GPU nodes and fail, while training pods sometimes get scheduled onto MIG-enabled nodes. What is the most appropriate way to configure the GPU Operator's MIG strategy so both workload types schedule correctly across the mixed fleet?

A cluster administrator needs to perform a firmware update on a GPU worker node in a Kubernetes cluster running NVIDIA GPU workloads. Several inference pods are currently scheduled on that node. The administrator wants to safely remove running pods and prevent new pods from being scheduled there before starting maintenance, while allowing the workloads to be rescheduled elsewhere. Which action should the administrator take?