Question

A data engineering team runs a daily AWS Glue Spark ETL job that reads from a large S3-based data lake containing several years of clickstream data partitioned by year, month, and day. The job only needs to process the most recent day of data, but it currently takes over an hour because the job scans the entire dataset before filtering. Which approach will most effectively reduce the amount of data read and improve job performance?

Accepted Answer

Apply a pushdown predicate on the partition columns when creating the DynamicFrame from the Data Catalog so only the relevant partitions are read from S3.

Answer

Increase the number of Glue DPUs allocated to the job so the full dataset is scanned faster in parallel.

Answer

Convert the source data from Parquet to JSON to make partition filtering easier for the Spark engine.

Answer

Add a Spark filter() transformation on the date column after loading the entire dataset into a DynamicFrame.