Question

A GenAI engineer deployed a customer-support RAG chatbot three months ago. Initial Mosaic AI Agent Evaluation runs on a golden dataset showed strong groundedness and correctness scores. Recently, users report increasingly stale or incorrect answers, though the chatbot code and prompts have not changed. The underlying knowledge-base Delta table (synced to the Vector Search index) is updated weekly by another team. The engineer wants an ongoing process to catch this kind of quality regression automatically rather than relying on user complaints. What is the most effective approach?

Accepted Answer

Schedule a recurring job that re-runs Mosaic AI Agent Evaluation against a maintained, periodically refreshed golden dataset and alerts when groundedness or correctness metrics drop below a defined threshold.

Answer

Enable Lakehouse Monitoring on the inference table and rely solely on request-volume and latency drift metrics to detect the quality regression.

Answer

Increase the retriever's top_k value so the chatbot always retrieves more chunks, reducing the chance of returning stale answers.

Answer

Re-log and re-deploy the chain weekly with pinned dependencies so the served model always matches the latest code version.