Question

A data scientist splits a Spark DataFrame into training and test sets using randomSplit([0.8, 0.2]). A colleague runs the exact same notebook code on the same cluster the next day but reports getting different rows in each split, causing model metrics to differ. What is the most likely cause, and how should it be fixed?

Accepted Answer

The split is not reproducible because no seed was provided; pass a fixed seed argument, e.g., randomSplit([0.8, 0.2], seed=42).

Answer

randomSplit always returns different results across sessions; switch to a pandas train_test_split, which is inherently deterministic.

Answer

The DataFrame must be cached before splitting; call df.cache() and the split will become reproducible without any seed.

Answer

The cluster autoscaled overnight; pin the cluster to a single worker so the split partitions identically each run.