Question

A data science team wants to explore several terabytes of raw event logs stored in Cloud Storage as Parquet files. They need to run ad hoc PySpark transformations and profiling from within their Vertex AI Workbench notebooks, but they do not want to provision or manage a persistent Spark cluster and want to minimize idle costs between exploration sessions. Which approach best meets these requirements?

Accepted Answer

Use Dataproc Serverless for Spark interactive sessions from the Vertex AI Workbench notebook to run PySpark against the Parquet data on demand.

Answer

Create a long-running Dataproc cluster with autoscaling enabled and attach it to the notebook via the Dataproc JupyterLab plugin.

Answer

Load the entire dataset into a pandas DataFrame in the notebook and process it with local Spark in standalone mode.

Answer

Export the Parquet files into BigQuery and run all exploration exclusively with BigQuery SQL, avoiding Spark entirely.