Question

A data scientist builds a Spark ML Pipeline consisting of a StringIndexer, a VectorAssembler, and a LogisticRegression estimator. After calling pipeline.fit(trainingDF), they want to persist the fitted result to DBFS so it can be reloaded later for scoring without re-running any training logic. Which object should they save, and how will it behave when reloaded?

Accepted Answer

Save the PipelineModel returned by pipeline.fit(); when reloaded, calling transform() applies the learned label indices and fitted coefficients directly to new data.

Answer

Save the original Pipeline object; when reloaded, calling transform() will apply the trained stages to new data.

Answer

Save only the LogisticRegression estimator; when reloaded, its transform() will re-index and score new data using the pipeline's other stages.

Answer

Save the PipelineModel; when reloaded, you must call fit() again on new data before transform() will produce predictions.