Question

A data engineering team must process thousands of small CSV files that arrive daily in an S3 prefix. Each file must be independently validated, cleansed, and written to a curated Parquet dataset by a Lambda function. The team wants to orchestrate this with AWS Step Functions while maximizing parallel processing and staying within Lambda concurrency limits, without manually splitting the file list. Which Step Functions approach best meets these requirements?

Accepted Answer

Use a Distributed Map state that reads the S3 object listing and invokes the Lambda function for each file with a configurable MaxConcurrency setting

Answer

Use a single Task state that invokes one Lambda function passing the entire list of file keys, letting the function loop through them sequentially

Answer

Use a Choice state that branches based on file count and invokes Lambda functions one at a time in a Wait loop

Answer

Use a Parallel state with a fixed number of hardcoded branches, each invoking the Lambda function on a subset of files