Question

A data engineering team stores 4 TB of tabular training data in Amazon S3 as gzip-compressed CSV files. Their SageMaker Processing jobs only need to read a subset of 12 columns out of 340 for feature engineering, and jobs are running slowly and incurring high I/O costs. The team wants to reduce the amount of data scanned per job while keeping storage in S3. What is the MOST effective change to the data storage format?

Accepted Answer

Convert the data to Apache Parquet, a columnar format that allows the processing jobs to read only the required columns.

Answer

Convert the data to uncompressed CSV so the processing jobs can seek directly to the needed columns without decompression overhead.

Answer

Convert the data to line-delimited JSON so each record is self-describing and columns can be selectively parsed.

Answer

Keep CSV but split each file into 340 separate single-column files, one per column, in a shared S3 prefix.