Question

Your team has a prototype trained in scikit-learn on a small sample of tabular data. The production dataset has 4 billion rows stored in BigQuery, with several high-cardinality categorical features. You must train a gradient-boosted model that scales to the full dataset, supports distributed training across multiple workers, and integrates cleanly with Vertex AI managed training. Which approach best meets these requirements?

Accepted Answer

Rewrite the model using XGBoost with its distributed (Dask or Spark) backend, run it as a Vertex AI custom training job across multiple workers, and read data in shards from BigQuery.

Answer

Continue with scikit-learn's GradientBoostingClassifier on a single high-memory Vertex AI training VM, loading the data into a pandas DataFrame.

Answer

Export all 4 billion rows to a single CSV in Cloud Storage and train scikit-learn with joblib parallelism on one machine with many vCPUs.

Answer

Convert the tabular data to images and train a convolutional neural network on TPUs for maximum throughput.