Question

A data scientist trains a binary classifier to predict whether a customer will churn. Only 4% of customers in the dataset actually churn. The initial model reports 96% accuracy on the validation set, but the business complains that almost no real churners are being flagged. Which evaluation approach should the data scientist adopt to properly assess and compare model performance for this problem?

Accepted Answer

Evaluate the model using the area under the precision-recall curve (AUPRC) along with recall for the churn class, because these metrics focus on performance on the rare positive class.

Answer

Continue optimizing for overall accuracy, since 96% indicates the model generalizes well to the validation set.

Answer

Switch to root mean squared error (RMSE) to measure the distance between predicted and actual churn labels.

Answer

Increase the size of the training dataset until accuracy exceeds 99%, which will automatically improve churn detection.