Question

Your team runs a persistent Dataproc cluster that executes a mix of Spark batch jobs. Job volume is highly variable: near-idle overnight, then heavy bursts during business hours. Finance wants to reduce cost without manual resizing, but jobs must complete reliably and HDFS data on the cluster must never be lost during scale-down events. What is the most appropriate way to configure the cluster?

Accepted Answer

Apply a Dataproc autoscaling policy that scales secondary (preemptible/non-preemptible) workers only, keeping a fixed pool of primary workers, with a graceful decommission timeout.

Answer

Apply an autoscaling policy that scales both primary and secondary workers aggressively down to zero primary workers during idle periods.

Answer

Write a Cloud Scheduler cron job that calls the Dataproc API to resize the cluster to a fixed large size at 8 AM and a small size at 8 PM.

Answer

Disable autoscaling and instead rely on Spark dynamic allocation alone to manage resources across the fixed cluster.