Question

A data engineering team uses an AWS Lambda function triggered by S3 event notifications to transform incoming CSV files into Parquet. The transformation works fine for most files, but recently some vendor files have grown to 8 GB each, and the Lambda invocations fail. CloudWatch logs show the function hitting its execution time and memory ceilings. The team wants the most appropriate solution to reliably transform these large files without re-architecting the entire event-driven flow. What should they do?

Accepted Answer

Have the S3 event trigger a Lambda that starts an AWS Glue ETL job to perform the transformation, letting Lambda act only as the orchestrator.

Answer

Increase the Lambda memory allocation to the 10 GB maximum and set the timeout to 15 minutes to accommodate the larger files.

Answer

Configure Lambda provisioned concurrency and split the file into smaller chunks using Lambda destinations.

Answer

Enable Lambda ephemeral storage (/tmp) up to 10 GB so the entire 8 GB file can be downloaded and processed in place.