Question

A financial services company is validating a custom fraud-detection model that will be integrated into a Dynamics 365 Finance agent. Fraudulent transactions represent only 0.8% of the training and test data. During review, the data science team reports 99.2% overall accuracy and wants to sign off on the model for production. As the solutions architect defining validation criteria, what should you require before approving the model?

Accepted Answer

Require validation against recall and precision (or F1) for the fraud class, because overall accuracy is misleading on a highly imbalanced dataset.

Answer

Approve the model, since 99.2% accuracy comfortably exceeds the typical 95% enterprise acceptance threshold.

Answer

Require the team to rebalance the dataset to 50/50 fraud and non-fraud and re-report accuracy on the rebalanced set.

Answer

Require only that inference latency stay under the agent's response SLA, since detection quality is the data science team's responsibility.