Question

Your team runs a nightly Spark ETL job on a persistent Dataproc cluster that sits idle most of the day. The job is fault-tolerant and can rerun if a task fails. Finance asks you to significantly cut the cluster's compute cost without missing the nightly SLA. Which approach best optimizes cost while meeting these requirements?

Accepted Answer

Create an ephemeral cluster per job run using a workflow template, sizing primary workers for the job and adding secondary preemptible (Spot) workers for scale-out.

Answer

Keep the persistent cluster running 24/7 but reduce the primary worker machine type to the smallest available size.

Answer

Move the workload to a single high-memory VM running Spark in standalone mode to avoid cluster management overhead.

Answer

Keep the persistent cluster and enable autoscaling so it scales primary workers down to zero when idle.