Question

A data scientist is building a RandomForestClassifier in a Databricks notebook and needs the entire pipeline to be fully reproducible across reruns. They already pass a fixed integer to train_test_split's random_state. On repeated runs, the train/test split is identical, but the trained model still produces slightly different predictions each time. What is the most likely cause and correct fix?

Accepted Answer

The RandomForestClassifier itself introduces randomness (bootstrap sampling and feature selection); set its random_state parameter to a fixed integer as well.

Answer

The train/test split seed must also be applied to the DataFrame's index before splitting; reset the index with a fixed seed.

Answer

MLflow autologging randomizes model outputs; disable autologging to obtain deterministic predictions.

Answer

Spark's lazy evaluation reorders rows nondeterministically; call .cache() on the training data to freeze the order.