Question

A financial services company deploys a Bedrock-based document classification application that returns a category label along with a confidence score (0.0–1.0) for each prediction. Downstream automated workflows only auto-approve documents when the model reports confidence above 0.9; everything else is routed to human reviewers. During validation, the QA team notices that many predictions scored above 0.9 are actually wrong, causing incorrect auto-approvals. They want a metric that specifically quantifies how well the model's reported confidence scores match the actual likelihood of correctness, so they can decide whether to trust the threshold. Which evaluation approach BEST addresses this need?

Accepted Answer

Compute a calibration metric such as Brier score or Expected Calibration Error (ECE) that compares predicted confidence against observed correctness across confidence bins.

Answer

Measure classification accuracy on the golden dataset and raise the auto-approval threshold to 0.95 if accuracy is below target.

Answer

Run a Bedrock model evaluation job using the built-in toxicity and robustness metrics to detect unreliable outputs.

Answer

Use an LLM-as-a-judge to re-score each classification and replace the model's confidence score with the judge's rating.