Question

A data engineering team stores raw ML training data across three S3 prefixes, each containing CSV files with different but overlapping schemas. Analysts need to run ad hoc Athena queries to explore and validate the data before feature engineering, and the team wants to minimize the manual effort of maintaining table definitions as new files arrive daily. Which approach best meets these requirements?

Accepted Answer

Configure AWS Glue crawlers to crawl each S3 prefix on a daily schedule, populating the Glue Data Catalog with table definitions that Athena can query.

Answer

Manually create AWS Glue Data Catalog table definitions for each prefix and update them with a Lambda function whenever new files are uploaded.

Answer

Load all CSV files into an Amazon Redshift cluster and run exploratory queries directly against Redshift tables.

Answer

Use Amazon SageMaker Data Wrangler to import each prefix individually and export inferred schemas to a JSON file for reference.