Question

A data engineering team runs a nightly Python cleanup script that deletes stale objects from Cloud Storage and updates a small metadata table in BigQuery. The script currently runs on a manually started Compute Engine VM that someone must remember to start and stop, and the team frequently forgets, leaving the VM running and incurring cost. The team wants a fully managed, serverless approach that runs the script on a fixed schedule with no idle infrastructure and minimal operational overhead. Which approach best meets these requirements?

Accepted Answer

Deploy the script as a Cloud Function (2nd gen) and invoke it on a cron schedule using Cloud Scheduler.

Answer

Keep the Compute Engine VM but add an instance schedule to start it nightly, and rely on the script to shut the VM down when finished.

Answer

Create a Cloud Composer environment and build a single DAG that runs the script daily via a PythonOperator.

Answer

Package the script into a Dataproc initialization action and run it on an ephemeral cluster each night.