Question

A data engineering team stores raw CSV files in an Amazon S3 data lake. They need to periodically clean the data, convert it to a columnar format for efficient querying, and catalog the resulting schema so it can be queried by Amazon Athena. The team wants a fully managed, serverless solution that requires no infrastructure to provision or manage. Which approach best meets these requirements?

Accepted Answer

Use AWS Glue crawlers to populate the Data Catalog and AWS Glue ETL jobs to transform the CSV files into Apache Parquet, writing the results back to S3.

Answer

Launch an Amazon EMR cluster with Apache Spark to run transformation jobs on a schedule and store metadata in an external Hive metastore on RDS.

Answer

Stream the CSV files through Amazon Kinesis Data Streams and use a Lambda consumer to convert records before writing them to S3.

Answer

Load the CSV files into an Amazon Redshift cluster, run SQL transformations, and unload the columnar results back to S3.