Question

A media company drops variable numbers of raw JSON log files into an S3 landing bucket at unpredictable times throughout the day. A data engineer must run an AWS Glue ETL job to clean and convert each new batch to Parquet within a few minutes of arrival, without incurring cost when no files are present. Which approach best meets these requirements?

Accepted Answer

Configure an EventBridge rule that matches S3 'Object Created' events and targets a Step Functions state machine that starts the Glue ETL job.

Answer

Schedule the Glue ETL job to run every 5 minutes using a Glue time-based trigger so new files are always picked up quickly.

Answer

Enable a Glue crawler on a 5-minute schedule and use an on-demand Glue trigger conditioned on crawler completion to start the ETL job.

Answer

Configure S3 to send a notification to an SQS queue and poll the queue continuously from a long-running EMR cluster that launches the Glue job.