Question

A data engineer needs to load historical order data stored as Parquet files in an S3 directory that is partitioned by year and month (e.g., /orders/year=2023/month=05/). The team's dashboard queries almost always filter on a specific year and month. When writing the batch read into a DataFrame, what should the engineer do to ensure Spark can leverage partition pruning and avoid scanning the entire dataset?

Accepted Answer

Read the base path /orders/ with spark.read.format("parquet") and rely on filters like where("year = 2023 and month = 5") so Spark discovers the partition columns and prunes at scan time

Answer

Read each subfolder individually by hardcoding the full path /orders/year=2023/month=05/ every time a different month is needed

Answer

Load the entire dataset into a DataFrame, cache it in memory, and let the cache serve all filtered queries

Answer

Convert the Parquet files to CSV first because CSV supports predicate pushdown more efficiently than Parquet