Question

A data engineer runs a PySpark notebook that reads a large Delta table into a DataFrame, then performs five separate aggregation actions on that same DataFrame within a loop. The Spark UI shows that the source Delta table is being read from OneLake and re-scanned during every action, causing the job to run much longer than expected. Which change will most effectively reduce the redundant reads and improve performance?

Accepted Answer

Persist the DataFrame in memory by calling cache() (or persist()) before the loop so subsequent actions reuse the materialized data

Answer

Increase the number of shuffle partitions with spark.sql.shuffle.partitions to parallelize the reads

Answer

Run OPTIMIZE with Z-ORDER on the source Delta table before executing the notebook

Answer

Enable Adaptive Query Execution (AQE) so Spark coalesces partitions automatically