Question

A data engineering team must process 12 million small JSON files landing daily in an S3 prefix. Each file requires an independent Lambda transformation, and the team needs the orchestration layer to fan out across all files with high concurrency while tracking success and failure of each item. The standard Map state fails because it holds the entire array in memory and caps at 40 concurrent iterations. Which approach best meets these requirements with minimal custom code?

Accepted Answer

Use a Step Functions Distributed Map state configured with an S3 object list as the item source, which processes items in batches with up to 10,000 parallel executions

Answer

Use a standard Map state and increase the MaxConcurrency parameter to 10,000 to process all files simultaneously

Answer

Trigger a single Lambda function with a 15-minute timeout that loops through all 12 million files sequentially

Answer

Configure an EventBridge rule to invoke the Lambda per S3 PutObject event and remove Step Functions entirely