Question

A data engineer builds an AWS Glue ETL job that reads semi-structured JSON order records from S3 and writes them to a curated Parquet dataset used to train a demand-forecasting model. Some records store the field 'quantity' as an integer (e.g., 5) while others store it as a string (e.g., "5"). When the job writes to Parquet, Glue creates two separate columns, 'quantity_int' and 'quantity_string', causing the downstream training job to fail. Which approach resolves this schema conflict most efficiently within the Glue job?

Accepted Answer

Apply the ResolveChoice transform with the 'cast' action to force the 'quantity' field to a single numeric type before writing to Parquet.

Answer

Enable Glue job bookmarks so that only new records with a consistent schema are processed on each run.

Answer

Re-run the Glue crawler with schema evolution enabled to merge the conflicting columns automatically.

Answer

Drop the 'quantity' column using the DropFields transform and recompute it during model training.