Question

A data engineer builds a Spark Structured Streaming job in a Fabric notebook that reads IoT sensor readings from an Eventstream into a Delta table. The requirement is to compute a 10-minute tumbling-window average temperature per device. Sensor events can arrive up to 5 minutes late, and the team wants Spark to bound the amount of in-memory state it must keep for aggregation while still admitting reasonably late events. Which approach correctly implements this?

Accepted Answer

Apply withWatermark on the event-time column with a 5-minute delay threshold, then group by a 10-minute window and device to aggregate the average

Answer

Group by a 10-minute window and device without a watermark, and set spark.sql.streaming.statefulOperator.stateTimeout to 5 minutes

Answer

Use foreachBatch to manually filter out events older than 5 minutes before writing each micro-batch to the Delta table

Answer

Set the trigger interval to 5 minutes so late events are naturally included, then group by device only and compute a running average