Question

A data engineering team runs dozens of ephemeral Dataproc clusters daily for Spark ETL jobs. Each cluster currently uses an initialization action script that installs a large set of Python libraries and a proprietary JDBC driver at boot time. Users complain that cluster provisioning now takes over 8 minutes, most of which is spent downloading and compiling dependencies. The team wants to reduce cluster startup time while keeping the exact same software stack for every job. What should they do?

Accepted Answer

Build a custom Dataproc image that pre-installs the libraries and JDBC driver, then create clusters from that image.

Answer

Move the initialization action script from Cloud Storage to a local file on the master node to speed up downloads.

Answer

Increase the number of secondary (preemptible) workers so dependency installation runs in parallel across more nodes.

Answer

Switch to a single-node Dataproc cluster so the initialization action only executes once.