Question

A data engineer is transforming a raw customer dataset in a Fabric notebook using PySpark before loading it into a lakehouse silver layer. The 'annual_income' column has approximately 8% null values that must be filled with the median income for downstream machine learning, while rows missing the 'customer_id' primary key must be discarded entirely. Which combination of operations correctly implements these two requirements?

Accepted Answer

Use df.na.drop(subset=['customer_id']) to remove rows with a null key, then use Imputer with strategy='median' on 'annual_income' to fill the remaining nulls.

Answer

Use df.na.fill(0, subset=['customer_id']) to replace null keys, then df.na.fill(df.agg(avg('annual_income')).first()[0], subset=['annual_income']).

Answer

Use dropDuplicates(['customer_id']) to remove null keys, then Imputer with strategy='mean' on 'annual_income'.

Answer

Use df.na.drop('all') to remove incomplete rows, then df.na.fill(df.approxQuantile('annual_income', [0.5], 0.0)) across all columns.