Question

A data engineering team maintains a data-cleansing job written entirely in Python using pandas and NumPy. The logic performs record-level normalization on datasets that have grown to several hundred gigabytes, and the single-node script now runs out of memory and takes hours. The team wants to scale the existing Python/pandas-style code across multiple nodes with minimal rewriting, staying within AWS Glue. Which approach best meets these requirements?

Accepted Answer

Use an AWS Glue for Ray job, which distributes Python and pandas-style workloads across a cluster of worker nodes.

Answer

Rewrite the entire job in PySpark DataFrame APIs and run it as a standard AWS Glue Spark ETL job.

Answer

Run the existing script as an AWS Glue Python shell job and increase the allocated DPUs to the maximum.

Answer

Migrate the pandas code to an AWS Lambda function with the maximum memory allocation and process the data in chunks.