Question

A data engineer is building an AWS Glue Spark ETL job to process raw event logs. Each incoming JSON record contains an array field named 'items' with a variable number of nested purchase objects. The business requires that each element of the 'items' array become its own separate row in the output Parquet dataset, with the parent record's metadata (order_id, customer_id, timestamp) repeated on every resulting row. Which transformation approach correctly produces the desired output?

Accepted Answer

Use the Spark DataFrame explode() function on the 'items' column after selecting the parent metadata columns, so each array element generates a distinct row.

Answer

Apply the Glue DynamicFrame ResolveChoice transform to cast the 'items' array to a struct and write directly to Parquet.

Answer

Use the Glue DropFields transform to remove the 'items' array and write only the parent metadata to Parquet.

Answer

Apply the Spark distinct() function to the DataFrame to deduplicate the nested items before writing to Parquet.