Question

A data scientist has trained a scikit-learn model on a single node and registered it with MLflow. They now need to score 500 million rows stored in a Spark DataFrame. They want to apply the model's predictions column-by-column using vectorized operations that process batches of rows as pandas Series for maximum throughput, without grouping the data. Which approach best fits this requirement?

Accepted Answer

Wrap the model in a scalar pandas UDF (pandas_udf) that accepts and returns a pandas Series, and apply it as a column in a Spark select/withColumn call

Answer

Use groupBy().applyInPandas() to distribute the model over grouped partitions of the DataFrame

Answer

Collect the Spark DataFrame to the driver with toPandas() and call model.predict() on the resulting pandas DataFrame

Answer

Convert the Spark DataFrame to an RDD and use map() to call model.predict() on each individual row