Question

A data engineer is preparing a tabular dataset of 5 million customer transactions stored as Parquet in Amazon S3 for a binary classification model. The team wants reproducible train/validation/test splits so that the exact same records land in each partition every time the preparation pipeline runs, regardless of the order in which Glue reads the files. Which approach best guarantees deterministic, reproducible splits?

Accepted Answer

Apply a hash function to a stable unique key column (e.g., transaction_id) and assign records to splits based on hash-bucket ranges.

Answer

Use Spark's randomSplit with default settings inside the Glue job so each run produces a fresh random partition of the data.

Answer

Sort the DataFrame by ingestion timestamp and take the first 70% for training, next 15% for validation, and last 15% for testing.

Answer

Let SageMaker automatically shuffle and split the data at training time using its built-in validation channel.