Implement generative AI quality assurance and observability

Your team built a generative AI summarization assistant in Azure AI Foundry. Reviewers complain that some generated summaries are grammatically awkward and read poorly, while others jump between ideas without logical flow. You want to set up an automated evaluation run that specifically quantifies (1) how grammatically correct and natural the language is, and (2) how well sentences connect and follow a logical structure. Which two built-in AI quality metrics should you configure?

Your team runs an Azure AI Foundry customer-support agent that must always include a legal disclaimer phrase in responses that mention refund policies. The built-in AI-assisted quality metrics (groundedness, relevance, coherence, fluency) do not capture this business rule. You need the automated batch evaluation workflow to flag every response missing the required phrase, in addition to the built-in metrics. What should you do?

Your team runs automated evaluation on a customer-support agent in Azure AI Foundry. The built-in groundedness, relevance, and coherence evaluators are already configured, but the business requires a domain-specific check: each response must include a valid support ticket ID in a fixed format, and the evaluation must report the percentage of responses that pass this check across the test dataset. You need to add this measurement to the existing automated evaluation workflow with the least ongoing maintenance. What should you do?

Your team is building an automated evaluation workflow in Azure AI Foundry for a customer-support chat agent. You have a test dataset containing the user query, the retrieved context passages, and the model's generated response, but you do NOT have any human-written ground-truth answers. Leadership wants a built-in AI quality metric that assesses whether the generated responses are grammatically correct and read naturally, independent of any reference answer. Which built-in metric should you configure?

Your team runs a RAG-based customer support agent in Azure AI Foundry. During evaluation, stakeholders report that the agent frequently produces answers that are well-written and directly address the user's question, but sometimes include facts that are NOT present in the retrieved documents. You need to configure an automated evaluation that specifically detects this problem while ignoring writing style. Which built-in AI quality metric should you prioritize?

Your team is preparing to evaluate a customer-support RAG agent in Azure AI Foundry before production release. You have only a small set of real user questions and need a larger, diverse test dataset that includes adversarial and edge-case queries so that safety and quality evaluators produce statistically meaningful results. You want to minimize the manual effort of hand-writing hundreds of prompts. Which approach best generates the required test dataset?

Your team is configuring an automated evaluation run in Azure AI Foundry for a RAG-based customer support agent. You have a JSONL test dataset whose columns are named 'user_question', 'retrieved_docs', and 'model_reply'. You want to run the built-in groundedness and relevance evaluators, but the evaluation run fails because the evaluators cannot find the expected inputs. What should you do to resolve this without renaming the dataset columns?

Your team built a multi-step agent in Azure AI Foundry that calls several tools (a search API, a calculator, and a database query function) to answer user requests. During evaluation, users report that the agent frequently selects the wrong tool or passes malformed arguments, even when the final text answer sounds plausible. You need to add an automated evaluation that specifically measures whether the agent chooses appropriate tools and invokes them with correct parameters, so you can gate deployment on this dimension. Which built-in evaluator should you add to your evaluation run?

Your team ships a customer-support agent in Azure AI Foundry that uses multiple tools to resolve billing questions. QA reports that the agent frequently misunderstands what the customer is actually asking for before it even selects a tool, leading to irrelevant tool calls. You must add an agent-specific evaluator to your automated evaluation run that directly measures how well the agent identifies the user's request from the conversation. Which built-in evaluator should you add?

Your team has deployed a customer-support RAG agent to production in Azure AI Foundry. Leadership wants ongoing assurance that live responses remain grounded and relevant without manually assembling a test dataset each time. You need an approach that automatically evaluates a sample of real production traffic on a recurring basis and surfaces the AI quality metrics in the monitoring dashboard. Which approach should you configure?

Your team has deployed a customer-support agent to production in Azure AI Foundry. Leadership is concerned about unpredictable monthly spend and wants a dashboard that shows how many tokens each conversation consumes over time, without writing custom instrumentation code. Which Foundry capability should you enable to satisfy this requirement?

Your team runs a multi-agent customer-support application in Azure AI Foundry. Finance reports that monthly token spend has doubled, but overall request volume is unchanged. You need to identify which specific agent operations and prompts are consuming the most tokens so you can optimize them, without disrupting production traffic. Which approach in Foundry observability should you use first?

A team has deployed a customer-support RAG agent to production in Azure AI Foundry. They already stream production interactions to a sampled online evaluation that computes groundedness and relevance on a percentage of live traffic. Management now wants to be automatically notified when the agent's groundedness score deteriorates in production so engineers can investigate before customers are widely affected. Without building any custom code, what should the team configure to meet this requirement?

Your team is preparing a customer-facing chatbot for release in Azure AI Foundry. Compliance requires that, before deployment, you quantify the application's likelihood of producing hateful, violent, sexual, and self-harm content when adversarially prompted. You want an automated evaluation that simulates adversarial user inputs and scores the responses against these harm categories. Which evaluation approach should you configure?

Your team runs a customer-facing chat agent deployed in Azure AI Foundry. Users complain that responses occasionally feel slow, but the overall average response time on your dashboard looks acceptable. Leadership wants to confirm that the agent meets a service-level objective stating that 95% of requests must complete within 3 seconds. Which observability approach in Foundry best validates whether this SLO is being met?

You operate a generative AI chat agent deployed in Azure AI Foundry. Support engineers report intermittent failures that are hard to reproduce, and you need to enable end-to-end diagnostics so that individual request traces, tool call spans, and exceptions can be queried and correlated with production incidents over the past 30 days. Which action best establishes this observability capability with minimal custom code?

Your team has deployed a multi-step generative AI agent in Azure AI Foundry that chains a retrieval tool, a summarization LLM call, and a final response-generation LLM call. Users report intermittent high end-to-end latency, but aggregate latency dashboards do not reveal which stage is responsible. You need to identify the specific step contributing to the slowdown for individual problematic requests without adding custom timing code to every function. What should you do?

A production RAG agent deployed in Azure AI Foundry is occasionally returning incomplete answers. You need to determine, for individual failing requests, exactly which retrieval tool call returned empty context and how the model's inputs and outputs flowed through each step. You have already enabled tracing on the agent. Which action lets you pinpoint the problematic step for a specific request?

Your team operates a Retrieval-Augmented Generation (RAG) chatbot deployed in Azure AI Foundry. Users complain that the assistant sometimes fabricates facts that are not present in the retrieved documents. You want to configure an automated evaluation run that specifically quantifies how well each generated answer is supported by the retrieved source context. Which built-in AI quality metric should you select?

Your team runs an automated evaluation flow in Azure AI Foundry for a customer-support RAG chatbot. Stakeholders complain that some answers are factually consistent with the retrieved documents but do not actually address what the user asked. You need to add a built-in AI quality metric to the evaluation that specifically measures how well the generated answer addresses the user's original question, independent of whether the answer is supported by the retrieved context. Which metric should you configure?