Question

A data engineering team runs a large nightly Spark ETL job on Dataproc. To reduce costs, they configured the cluster to use secondary workers that are Spot VMs. The job frequently fails during the shuffle-heavy stages because preempted workers lose intermediate shuffle data, forcing expensive recomputation and sometimes exceeding retry limits. The team wants to keep the cost savings of Spot VMs while making the job resilient to preemption. What should they do?

Accepted Answer

Enable Enhanced Flexibility Mode (EFM) so shuffle data is stored on primary workers rather than on preemptible secondary workers.

Answer

Convert all secondary Spot workers to standard on-demand secondary workers to eliminate preemption entirely.

Answer

Increase the number of Spark task retries and lengthen the executor heartbeat timeout in the job configuration.

Answer

Move the shuffle data to a Cloud Storage bucket by setting the Spark local directory to a GCS path.