Question

A data engineering team runs a nightly Cloud Run job that processes 10,000 independent CSV files from Cloud Storage. Each file takes about 30 seconds to process. Currently the job runs with a single task, taking over 80 hours—well past the nightly window. The team wants to dramatically reduce wall-clock time while keeping cost predictable and avoiding wasted idle compute. Which configuration change best meets these goals?

Accepted Answer

Increase the Cloud Run job's task count to shard the workload and raise task parallelism so many files are processed concurrently, sizing CPU/memory per task to the per-file need.

Answer

Convert the job to a Cloud Run service with min-instances set high so it stays warm and processes files as HTTP requests arrive.

Answer

Increase the CPU and memory of the single task to the maximum allowed so each file processes faster sequentially.

Answer

Migrate the workload to a large Compute Engine VM with a startup script that loops through all files sequentially.