Question

A developer builds a Lambda function triggered by S3 object-created events. Each uploaded object is a large CSV file (up to 500 MB) that must be parsed and its rows written to DynamoDB. During testing, the function occasionally fails with memory errors and sometimes times out. Which approach BEST addresses processing these large files reliably while following AWS best practices?

Accepted Answer

Stream the S3 object using range GET requests to read it in chunks, increase the Lambda memory and timeout appropriately, and process rows in batches so the whole file is never loaded into memory at once.

Answer

Configure the S3 event to pass the entire file contents as the event payload so Lambda never needs to download the object from S3.

Answer

Set Lambda memory to the maximum of 128 MB and rely on automatic retries to eventually complete processing of the file.

Answer

Enable provisioned concurrency on the function so that large files are processed faster and avoid timeouts.