Question

A data scientist has been prototyping feature engineering using a pandas DataFrame in a Databricks notebook. The dataset has grown to 400 GB and no longer fits in the driver's memory, causing out-of-memory errors. The scientist wants to keep the existing pandas-style code (e.g., df.groupby(...).mean(), df['col'].fillna(...)) with minimal rewriting while distributing the computation across the cluster. Which approach best meets this requirement?

Accepted Answer

Import pyspark.pandas as ps and load the data with ps.read_parquet, keeping the pandas-like syntax while executing on Spark

Answer

Increase the driver node instance size so the 400 GB DataFrame fits in memory and continue using pandas unchanged

Answer

Convert the pandas code to native PySpark DataFrame API calls, rewriting every transformation using select, groupBy, and agg

Answer

Wrap the pandas functions in a pandas UDF and apply them column-by-column to a Spark DataFrame