Question

A data engineer runs a nightly Spark notebook that reads a large Delta table in a lakehouse. The table stores three years of transactions and is partitioned by a 'transaction_date' column. Analysts only need the most recent 7 days of data, but the notebook currently loads the full DataFrame with spark.read and then applies a .filter() on transaction_date after several joins and aggregations. Runtime has grown to over 45 minutes. What is the MOST effective change to reduce the amount of data scanned?

Accepted Answer

Apply the transaction_date filter immediately after reading the table so partition pruning eliminates irrelevant partition folders before joins execute

Answer

Increase the number of Spark executors in the environment to process the full three years of data faster

Answer

Cache the full DataFrame in memory with .cache() before applying the joins and filter

Answer

Convert the Delta table to CSV format so the reader can scan files more quickly