Question

You are validating a Copilot Studio agent used by a retail contact center. During test execution, the QA team complains that automated regression tests are unreliable: the same test input for an order-status inquiry produces slightly different natural-language responses on each run, causing exact-string assertions to fail even when the agent behaves correctly. The underlying tool calls and retrieved order data are consistent. How should you redesign the test assertion approach to produce a reliable, repeatable validation process?

Accepted Answer

Assert on deterministic signals such as the correct tool/action being invoked with expected parameters and the presence of key facts in the response, using semantic similarity or keyword checks rather than exact-string matching.

Answer

Set the agent's model temperature to zero in production so that responses become fully deterministic and exact-string assertions always pass.

Answer

Remove automated regression tests for generative responses entirely and rely only on manual reviewer spot-checks before each release.

Answer

Freeze the model version and hard-code the exact expected response text captured from the first successful test run as the golden output.