Question

A data engineer builds a Structured Streaming query that reads new files from cloud storage using Auto Loader and writes to a Delta table. The business only needs the data refreshed once every hour, and the team wants to minimize compute costs by spinning down the cluster between runs rather than keeping it running 24/7. Which approach best satisfies these requirements?

Accepted Answer

Configure the streaming query with .trigger(availableNow=True) and run it on an hourly scheduled job that terminates the cluster when finished.

Answer

Configure the streaming query with a default (continuous micro-batch) trigger and keep an all-purpose cluster running so it processes data as it arrives.

Answer

Configure the streaming query with .trigger(processingTime='1 hour') on an always-on cluster so a micro-batch fires every hour.

Answer

Rewrite the pipeline as a static batch read with spark.read and append to the Delta table, scheduled hourly.