Hard NCA-AIIO practice questions

During routine fleet monitoring, DCGM reports a steadily increasing count of correctable (single-bit) ECC memory errors on one GPU in a training node. The GPU continues to complete jobs successfully, but the error count grows week over week. What is the most appropriate operations response?

After a routine driver upgrade on a DGX system with NVSwitch, several multi-GPU training jobs fail immediately at startup with NCCL errors, and nvidia-smi shows all GPUs present and healthy. Single-GPU jobs run fine. Checking the host, you find the fabric manager service is not running. What is the most likely cause and correct remediation?

A multi-GPU training job on a DGX node suddenly shows much lower throughput than a previous identical run on the same node. GPU utilization (SM activity) and power draw both read near 100%, temperatures are normal, and no XID errors appear in the logs. An administrator wants to confirm whether inter-GPU communication has degraded before escalating a hardware ticket. Which DCGM metric should the administrator examine first?

You administer a multi-node training cluster. Users report that a specific 8-GPU node occasionally produces corrupted gradient reductions during NCCL all-reduce operations, but only under heavy load. GPU utilization, power draw, and temperature all appear normal in your dashboards. You want to use DCGM to pinpoint whether an intermittent interconnect fault is the root cause. Which DCGM field is most appropriate to investigate first?

An AI infrastructure team is designing the compute fabric for a multi-node DGX H100 SuperPOD cluster. They want to maximize the performance of distributed training collective operations (like all-reduce) across many nodes while ensuring each GPU has a dedicated, predictable path into the InfiniBand fabric. Which network design approach should they adopt for the compute fabric?

A research organization is building an AI cluster of 32 DGX H100 nodes for large-scale distributed training. The network architect must design the compute fabric so that inter-node GPU-to-GPU collective operations (like all-reduce during gradient synchronization) are not the bottleneck. Which fabric design best meets this requirement?

An AI infrastructure team is deploying a multi-rack GPU cluster for large-model training. They need shared, high-throughput access to a multi-petabyte dataset from every compute node while ensuring dataset reads never compete for bandwidth with the GPU-to-GPU gradient synchronization traffic. Which infrastructure design best meets this requirement?

A team is deploying a multi-node GPU training cluster in a public cloud region that does not offer InfiniBand, but does offer instances with RDMA over Converged Ethernet (RoCE) support at 400 Gb/s per node. The training job is a large distributed model that relies heavily on frequent all-reduce collective operations across nodes. Which statement best describes the appropriate networking approach for this deployment?

A machine learning team currently trains a large model on a single 8-GPU node, and one full training epoch takes roughly 24 hours. Business stakeholders now require the same model to complete an epoch in about 3 hours to accelerate experimentation. Assuming the workload scales well across nodes and the interconnect is not the bottleneck, which infrastructure sizing decision best meets this target?

A research team runs a computational fluid dynamics simulation that requires high-precision double-precision (FP64) arithmetic. They plan to reuse an inference-optimized GPU that advertises very high FP8 and INT8 throughput but comparatively modest FP64 performance. Why might this GPU be a poor fit for their workload?

A data science team benchmarks a large language model on two GPUs. On the newer GPU, matrix-multiply-heavy training steps run dramatically faster when the framework enables FP8/mixed-precision math, but element-wise data-preprocessing kernels show almost no speedup. An engineer asks why the same GPU benefits enormously on one workload but not the other. What is the most accurate explanation?

A data scientist notices that a deep learning training job is running far slower than expected on a GPU server. Profiling reveals that during every training step, the GPU sits mostly idle while large volumes of data are repeatedly copied from system (CPU) memory across the PCIe bus to the GPU. Which fundamental GPU computing principle explains why this pattern degrades performance, and what should be optimized?

A research team is building a recommender system whose embedding tables are far too large to fit in GPU VRAM, forcing constant data movement between the CPU's large system memory and the GPU. They want a single platform that tightly couples a CPU and GPU with a high-bandwidth, cache-coherent link so the GPU can access the CPU's memory pool efficiently without traditional PCIe bottlenecks. Which NVIDIA platform best fits this requirement?

A data center team is deploying several NVIDIA HGX B200 8-GPU nodes into an existing colocation facility. Each rack in the facility is rated for a maximum of 30 kW, and the team wants to place as many GPU nodes per rack as thermal limits allow while keeping deployment simple. A single HGX B200 node draws roughly 14.3 kW at full load. The facility currently supplies only chilled-air cooling with no rear-door heat exchangers or direct-to-chip liquid loops available. Which action should the team take to deploy these nodes reliably?

A research team runs large-scale distributed training across 32 DGX nodes connected with NVIDIA Quantum-2 InfiniBand. Profiling shows that the all-reduce collective operations during gradient synchronization consume a growing share of each training step as they scale out. The infrastructure architect wants to reduce this communication overhead by offloading part of the reduction work from the GPUs and host CPUs. Which InfiniBand capability directly addresses this?

A platform engineer manages a Kubernetes cluster using the NVIDIA GPU Operator. After a rolling worker-node OS upgrade, GPU-requesting pods on the upgraded nodes stay in 'Pending' state, and `kubectl describe node` shows the nodes no longer advertise the `nvidia.com/gpu` extended resource. The nodes themselves are Ready, and other (non-GPU) pods schedule normally. What is the most likely root cause to investigate first?

You administer a Kubernetes cluster with the NVIDIA GPU Operator. Some A100 nodes are configured with MIG enabled to serve many small inference pods, while other A100 nodes remain in full-GPU (non-MIG) mode for large training jobs. Data scientists report that inference pods requesting a MIG slice occasionally land on the full-GPU nodes and fail, while training pods sometimes get scheduled onto MIG-enabled nodes. What is the most appropriate way to configure the GPU Operator's MIG strategy so both workload types schedule correctly across the mixed fleet?

A Kubernetes cluster has several A100 GPU nodes. One node has been partitioned so a single GPU exposes both a 3g.20gb slice and two 1g.5gb slices to serve differently sized inference models on the same physical GPU. Pods requesting the smaller MIG profile remain Pending on this node even though free 1g.5gb slices exist. Which cluster configuration is the most likely cause?

A research team is deploying a cluster to train a trillion-parameter model that requires very high all-to-all GPU bandwidth across 32 GPUs spanning four DGX H100 nodes. They want these 32 GPUs to behave as a single large scale-up domain with NVLink-class bandwidth between GPUs in different chassis, rather than relying on the node-internal NVSwitch fabric only. Which technology enables extending the NVLink domain across multiple physical nodes?

A systems engineer is configuring a PCIe-based server with four GPUs for a data-loading-intensive inference pipeline. During validation, GPUs 2 and 3 show significantly reduced host-to-device transfer bandwidth compared to GPUs 0 and 1, even though all four cards are identical. Reviewing the server topology, the engineer finds that GPUs 2 and 3 are connected through a PCIe switch that shares a single upstream x16 link to the CPU, while GPUs 0 and 1 each have dedicated x16 links. What is the most likely cause of the observed bandwidth difference?

A shared Slurm cluster has several A100 nodes. The platform team wants to give a group of inference users access to small, isolated GPU slices while training users continue to request whole GPUs on other nodes. They have enabled MIG on the inference nodes with 1g.10gb profiles. What must the administrator configure so Slurm can schedule jobs onto the individual MIG instances?