Question

A data scientist runs df.describe() on a Spark DataFrame of loan applications before modeling. For the 'annual_income' column, the summary shows: count = 48,500 (out of 50,000 rows), min = -1, mean = 62,300, 50% (median) = 55,000, max = 9,999,999. Which single observation from these statistics most strongly indicates a data quality problem that should be investigated before feature engineering?

Accepted Answer

The min value of -1 is present, since annual income cannot logically be negative.

Answer

The mean (62,300) is higher than the median (55,000), which proves the data is corrupted.

Answer

The count of 48,500 is fine because Spark automatically imputes the 1,500 missing values.

Answer

The max of 9,999,999 confirms the column must be dropped because it contains only outliers.