Question

Your team is migrating a set of recurring Spark ETL jobs from an on-premises Hadoop cluster to Google Cloud. The jobs currently run on a shared, always-on cluster, but each job has different memory and CPU requirements. Leadership wants to avoid a permanently running cluster, minimize idle cost, and ensure each job runs on infrastructure sized to its needs while keeping the orchestration definition version-controlled and repeatable. Which approach best meets these goals?

Accepted Answer

Define Dataproc Workflow Templates that use managed (ephemeral) clusters, specifying per-workflow cluster configurations, and store the templates in source control.

Answer

Create a single long-running Dataproc cluster sized for the largest job and submit all jobs to it sequentially.

Answer

Install open-source Hadoop on Compute Engine VMs and use cron on a bastion host to trigger each Spark job.

Answer

Migrate all jobs to a single BigQuery scheduled query and drop the Spark logic entirely.