Question

A data science team prototypes models in Vertex AI Workbench notebooks. Their preprocessing code uses pandas to read and transform a 400 GB BigQuery table, but the notebook kernel repeatedly crashes with out-of-memory errors. The team wants to keep writing familiar pandas-style code in the notebook while pushing the heavy computation down to BigQuery so the data never has to fit in the notebook's memory. What should they do?

Accepted Answer

Use the BigQuery DataFrames (bigframes) library, which exposes a pandas-compatible API but executes operations as BigQuery SQL on the server side.

Answer

Increase the Workbench instance to the largest available machine type with more RAM and continue using pandas with the BigQuery client.

Answer

Export the BigQuery table to CSV files in Cloud Storage and read them into pandas in chunks with the chunksize parameter.

Answer

Load the table into a Vertex AI Feature Store and read features into a pandas DataFrame for preprocessing.