Question

A Business Solutions Architect has deployed a custom AI model that scores incoming service cases for escalation risk within Dynamics 365 Customer Service. The model performed well in validation, but the operations team has no formal way to detect when it silently degrades in production. Business stakeholders want to be alerted before customers are affected, not weeks later during a quarterly review. Which monitoring approach should the architect implement to best detect real-world model degradation over time?

Accepted Answer

Configure continuous monitoring that tracks prediction distribution shifts and input feature drift against the training baseline, with automated alerts when metrics breach defined thresholds

Answer

Rely on user-submitted feedback tickets in the backlog and manually review model behavior only when complaint volume rises

Answer

Re-run the original validation test dataset against the production model on a fixed monthly schedule and compare precision and recall

Answer

Enable Dynamics 365 audit logging on the case entity so all field changes are recorded for later forensic review