Question

An analyst runs the same dashboard query dozens of times per hour against a large Delta table. Each execution rereads the same Parquet files from cloud object storage, and the Spark UI shows most of the stage time spent in remote reads rather than compute. The data changes only once per day after an overnight batch load. Which optimization will most directly reduce the repeated remote-read latency for these interactive queries?

Accepted Answer

Enable disk caching (Delta cache) on the cluster so frequently accessed Parquet files are stored on local SSDs of the worker nodes

Answer

Increase the number of shuffle partitions with spark.sql.shuffle.partitions to parallelize the reads

Answer

Run VACUUM on the table with a zero-hour retention to remove stale files and speed up scans

Answer

Repartition the DataFrame in memory before each query using repartition(200)