Question

A retail company is migrating a 200-node on-premises Hadoop cluster to Google Cloud. The cluster runs a mix of scheduled Spark batch jobs and ad-hoc Hive queries. Management wants to minimize long-term compute costs, decouple storage from compute so clusters can be shut down when idle, and avoid re-architecting the existing Spark and Hive code. Which migration approach best meets these requirements?

Accepted Answer

Migrate the data to Cloud Storage and run ephemeral, job-scoped Dataproc clusters that spin up per job and shut down on completion.

Answer

Store data in HDFS on persistent Dataproc clusters that run 24/7, and lift-and-shift the Spark and Hive jobs unchanged.

Answer

Rewrite all Spark jobs as Dataflow pipelines and migrate Hive queries to BigQuery before decommissioning the cluster.

Answer

Provision a single large Compute Engine VM with Hadoop installed and copy HDFS data onto its local persistent disks.