Question

A financial services team trains a gradient boosting classifier to predict loan default. Downstream systems consume the model's predicted probabilities directly as risk scores to set interest rates, so the numeric probabilities must reflect true likelihoods (e.g., a predicted 0.30 should default about 30% of the time). During evaluation, the model shows strong ranking performance with AUC of 0.88, but the business team reports the predicted probabilities appear systematically too high in the mid-range. Which evaluation step best addresses whether the probabilities can be trusted as risk scores?

Accepted Answer

Plot a calibration (reliability) curve of predicted probability versus observed frequency and apply probability calibration such as isotonic regression or Platt scaling if miscalibrated

Answer

Increase the classification threshold from 0.5 to 0.7 to reduce the number of high-probability predictions

Answer

Re-evaluate using AUC on a larger holdout set, since a higher AUC guarantees well-calibrated probabilities

Answer

Switch the evaluation metric to F1 score to balance precision and recall for the mid-range predictions