Question

Your team has a working prototype: a gradient-boosted tree model trained in a scikit-learn notebook on a 5 GB CSV file. Product wants this retrained weekly on a growing dataset (now 400 GB, stored in Cloud Storage as sharded Parquet files) with minimal custom infrastructure. The data scientists prefer to keep writing Python and want managed, repeatable training runs. Which approach best scales this prototype for production?

Accepted Answer

Rewrite the training as a distributed XGBoost or Spark MLlib job and run it on Vertex AI custom training (or Dataproc), reading Parquet directly from Cloud Storage and orchestrating weekly runs with a Vertex AI Pipeline.

Answer

Package the scikit-learn code into a custom training container and submit it as a single-replica Vertex AI custom training job on a large high-memory VM, loading all Parquet files into a pandas DataFrame in memory.

Answer

Keep the scikit-learn notebook, attach a larger GPU to the Workbench instance, and manually rerun the notebook each week.

Answer

Migrate the model to Vertex AI AutoML Tabular and retrain weekly, discarding the existing gradient-boosted tree logic.