Question

A data engineering team ingests semi-structured JSON event logs into an S3 bucket daily. The schema of these events evolves frequently — new optional fields appear and some fields change from single values to arrays. The team uses an AWS Glue crawler to populate a Data Catalog table that a downstream Glue ETL job reads to build ML training features. Recently, the crawler has been creating multiple separate tables and marking fields as incompatible, breaking the ETL job. What is the most appropriate way to handle this evolving schema so the ETL job continues to run reliably?

Accepted Answer

Configure the crawler to update the existing table and choose the option to merge new columns into the existing schema, while using Glue DynamicFrames with ResolveChoice in the ETL job to reconcile ambiguous field types.

Answer

Delete the Data Catalog table before each crawler run so a fresh schema is inferred every time from the latest data.

Answer

Disable the crawler and manually define a fixed schema in the Data Catalog that never changes, forcing all incoming data to conform.

Answer

Convert all incoming JSON to CSV before crawling so that Glue always infers a flat, consistent schema.