Question

A data scientist has a scikit-learn model that expects a pandas DataFrame and returns a prediction plus a probability score. They want to distribute batch inference across a large Spark DataFrame by grouping records by 'region' and applying the model to each group's data as a pandas DataFrame using groupBy().applyInPandas(). What must be provided for this to execute correctly on Spark?

Accepted Answer

An explicit output schema (StructType) describing the columns and types the applied function returns

Answer

A registered Unity Catalog model URI passed directly to applyInPandas()

Answer

A returnType argument specifying only a single scalar type such as DoubleType

Answer

A vectorized pandas_udf decorator applied to the grouping function