Question

A data engineering team runs an AWS Glue Spark ETL job that processes a large daily fact table (approximately 500 GB) of clickstream events stored in S3. Each event record must be enriched with product category and region attributes from a small reference dataset (about 40 MB) also stored in S3. The current job performs a standard join between the two DynamicFrames/DataFrames, but the job is slow and produces heavy shuffle activity across executors. Which approach will most effectively reduce shuffle and speed up the enrichment join?

Accepted Answer

Broadcast the small 40 MB reference dataset so it is replicated to all executors, allowing a map-side (broadcast) join with the large fact table

Answer

Repartition both datasets on the join key with a very high partition count before joining to force parallel processing

Answer

Convert both datasets to JSON before the join to reduce serialization overhead across the shuffle

Answer

Increase the number of DPUs (workers) so the shuffle completes faster across more executors