Question

A data science team prototypes models in a Vertex AI Workbench notebook using a 50 GB sample pulled into a Pandas DataFrame. The full training dataset lives in BigQuery and is roughly 8 TB, requiring complex windowed aggregations and joins across several tables. The team wants a preprocessing approach that scales to the full dataset for production training runs, minimizes custom infrastructure management, and produces feature tables that both training and future batch jobs can reuse. Which approach should they adopt?

Accepted Answer

Perform the set-based aggregations and joins directly in BigQuery using SQL, materializing the results into curated feature tables that downstream jobs query.

Answer

Increase the notebook machine to a high-memory instance and continue running the transformations with Pandas on the full 8 TB dataset.

Answer

Export the BigQuery tables to CSV files in Cloud Storage and process them with a manually configured Dataproc cluster running Spark.

Answer

Load the full dataset into the notebook in chunks and loop over each chunk with scikit-learn preprocessing transformers.