Question

You are training a large deep learning model in Azure Machine Learning using a PyTorch script that already implements DistributedDataParallel. You want to run the job across 4 nodes with 4 GPUs each on a GPU compute cluster, and you need Azure Machine Learning to launch one process per GPU and set the correct environment variables (RANK, WORLD_SIZE, MASTER_ADDR) for the distributed process group. How should you configure the command job?

Accepted Answer

Add a PyTorch distribution configuration with process_count_per_instance set to 4 and set the job's instance_count (resources) to 4.

Answer

Set instance_count to 16 with no distribution block, relying on the cluster to auto-detect the GPUs and spawn processes.

Answer

Add an MPI distribution configuration with process_count_per_instance set to 16 and set instance_count to 1.

Answer

Wrap the script in a ParallelRunStep with mini_batch_size tuned to the number of GPUs and set node_count to 4.