Question

A data scientist is building a Spark ML pipeline. They apply a StringIndexer to a categorical column, then feed the resulting index column into a OneHotEncoder before assembling features. During code review, a colleague asks why the OneHotEncoder must be called with .fit() on the training data rather than simply transforming rows directly. What is the correct explanation of OneHotEncoder's behavior in Spark ML?

Accepted Answer

In Spark ML, OneHotEncoder is an Estimator that must be fit to learn the number of distinct category indices, producing a OneHotEncoderModel (a Transformer) used to encode data.

Answer

OneHotEncoder is a pure Transformer that requires no fitting; calling .fit() is optional and has no effect on the output.

Answer

OneHotEncoder must be fit because it computes the mean and variance of each index to normalize the one-hot vectors.

Answer

OneHotEncoder is fit only so it can persist the original string labels; the encoding itself is deterministic and does not depend on training data.