Question

A media analytics company runs nightly Spark batch jobs on a self-managed Hadoop cluster on-premises. Leadership wants a cloud design that avoids vendor lock-in, so future workloads could potentially run on another cloud provider with minimal rewrites. The engineering team plans to use Dataproc but is concerned that tightly coupling their data and job logic to proprietary services will make a future move costly. Which design approach best preserves portability while running on Google Cloud?

Accepted Answer

Store data in Cloud Storage using open formats such as Parquet or ORC, keep Spark job logic in standard open-source APIs, and run on ephemeral Dataproc clusters.

Answer

Load all data into BigQuery and rewrite the Spark jobs as BigQuery SQL stored procedures to take advantage of the managed engine.

Answer

Persist data on HDFS attached to a long-running Dataproc cluster and use Dataproc-specific autoscaling policies for all processing.

Answer

Store data in Bigtable and refactor jobs to use the Bigtable client libraries for high-throughput reads and writes.