Question

A logistics company runs a nightly ETL job on a self-managed Hadoop cluster hosted on Compute Engine VMs. The cluster is only fully utilized for about 3 hours each night but runs 24/7, and the operations team spends significant time patching and tuning it. Leadership wants to cut both infrastructure spend and operational overhead while keeping the existing Spark and Hive jobs running with minimal code changes. Which approach best meets these business requirements?

Accepted Answer

Migrate the jobs to ephemeral Dataproc clusters that are created before each nightly job and deleted after completion, storing data in Cloud Storage.

Answer

Resize the existing Compute Engine cluster to smaller machine types and add a cron job to power the VMs off during idle hours.

Answer

Rewrite all Spark and Hive jobs as Dataflow pipelines running continuously on a managed streaming service.

Answer

Move the persistent Hadoop cluster to a committed use discount contract to lower the hourly compute rate.