Question

A data engineer is building an AWS Glue Spark ETL job that reads semi-structured JSON files from S3. The same field 'amount' appears as a string in some records and as a double in others, causing the job to fail when writing to a Parquet target with a fixed schema. The engineer wants to handle the ambiguous column so that all values are cast to a single numeric type without dropping records. Which approach best resolves this within the Glue job?

Accepted Answer

Apply the ResolveChoice transform on the DynamicFrame with a cast to 'double' for the 'amount' column before writing to Parquet

Answer

Convert the DynamicFrame to a Spark DataFrame immediately and rely on Spark's automatic schema inference to unify the types

Answer

Use the ApplyMapping transform to rename the 'amount' column so the conflicting types are stored separately

Answer

Enable a Glue job bookmark so that only records with consistent types are processed on each run