Question

A media analytics company runs dozens of independent Spark batch jobs each day on a single long-running Dataproc cluster. The team complains about noisy-neighbor resource contention, version conflicts between jobs requiring different Spark releases, and high costs from the cluster running 24/7 even when idle. They want a design that maximizes flexibility and portability while minimizing cost. What should you recommend?

Accepted Answer

Create an ephemeral Dataproc cluster per job using workflow templates, storing all input and output data in Cloud Storage, and delete each cluster when its job completes.

Answer

Increase the size of the single persistent cluster and use YARN queues to isolate the jobs from one another.

Answer

Migrate all Spark jobs to a persistent cluster with autoscaling enabled and store intermediate data on cluster-local HDFS.

Answer

Keep the persistent cluster but schedule jobs to run sequentially overnight to avoid resource contention.