Question

A data scientist is building a Spark ML pipeline and needs to combine three numeric columns ('tenure', 'monthly_charges', 'total_charges') into a single 'features' vector column that will feed a LogisticRegression stage. They plan to add a VectorAssembler as one of the pipeline stages. When the pipeline's fit() method is called on the training DataFrame, what happens with respect to the VectorAssembler stage specifically?

Accepted Answer

The VectorAssembler is a transformer that requires no fitting; it simply applies its transform() logic to concatenate the columns into the output vector.

Answer

The VectorAssembler learns column statistics during fit() and applies them during transform(), making it an estimator.

Answer

The VectorAssembler must be fit separately before being added to the pipeline because it cannot be nested inside a Pipeline object.

Answer

The VectorAssembler will raise an error during fit() unless the input columns are first indexed with a StringIndexer.