Question

A data engineering team lands compressed Avro files in an S3 bucket every hour from an upstream application. The Avro schema occasionally adds new optional fields over time. The team wants Amazon Athena to query the latest data without manual schema updates each time a new field appears, while minimizing operational overhead. Which approach best meets these requirements?

Accepted Answer

Configure an AWS Glue crawler to run on a schedule against the S3 path and enable the option to update the table definition in the Data Catalog when new columns are detected.

Answer

Manually define the Athena table with all current columns using a CREATE EXTERNAL TABLE statement and update the DDL each time a schema change occurs.

Answer

Convert the Avro files to CSV with a Lambda function before querying, so Athena can infer the schema automatically at query time.

Answer

Load the Avro files into an Amazon Redshift table using COPY and query them through Redshift Spectrum without cataloging the data.