Question

Your team runs a large nightly Dataflow batch pipeline that processes several terabytes of data. The job must complete before the 6 AM business-hours window but has no strict start time, and it is fully fault-tolerant with checkpointing. Finance is pushing to reduce the compute cost of this pipeline without adding operational overhead or missing the deadline. Which approach best meets these requirements?

Accepted Answer

Enable Flexible Resource Scheduling (FlexRS) on the Dataflow batch job so it uses a mix of preemptible and regular VMs and schedules within a delay window.

Answer

Convert the batch pipeline to a streaming pipeline with Streaming Engine to spread cost evenly across the day.

Answer

Manually launch the job on a fixed pool of committed-use n2 VMs sized for peak throughput to guarantee the deadline.

Answer

Disable autoscaling and set a small fixed number of workers to cap the maximum hourly spend.