Question

Your team has developed a PyTorch training prototype in a notebook. You need to move it to a scalable, repeatable training job on Vertex AI Custom Training. The training script depends on a few pip-installable libraries and a proprietary internal C++ library that must be compiled from source and is not available on PyPI. You want the most maintainable and reliable way to run this on Vertex AI. What should you do?

Accepted Answer

Build a custom container image that starts from a base PyTorch image, compiles and installs the C++ library plus the pip dependencies, push it to Artifact Registry, and submit it as a custom container training job.

Answer

Use a Vertex AI prebuilt PyTorch training container and pass the pip packages via the requirements argument, then install the C++ library at runtime from a startup script.

Answer

Package the training code as a Python source distribution and submit it with a prebuilt container, bundling the compiled C++ library binaries inside the Python package.

Answer

Convert the training code to run on BigQuery ML so no container or dependency management is required.