Evaluation and Monitoring — Databricks GenAI Engineer Associate Practice Quiz

Question 1

A Generative AI Engineer is running Mosaic AI Agent Evaluation on a RAG-based customer support assistant. The evaluation dataset includes a column with expert-written reference answers for each question. The team wants to measure how factually and semantically close the agent's generated responses are to these reference answers, independent of whether the retrieved context supported them. Which built-in Agent Evaluation metric directly addresses this requirement?

Accepted Answer

answer correctness, which compares the agent's response against the provided ground-truth reference answers

Answer

groundedness, which checks whether the response is supported by the retrieved context

Answer

chunk relevance (retrieval precision), which measures how relevant retrieved documents are to the request

Answer

safety, which flags whether the response contains harmful or toxic content

Question 2

A team runs Mosaic AI Agent Evaluation nightly on a golden dataset of 500 questions for their production RAG assistant. After a recent change that swapped the generation model to a larger foundation model, the evaluation report shows the average correctness and groundedness scores each improved by about 3%, but the total_input_token_count and total_output_token_count metrics roughly tripled, and mean response latency rose from 1.2s to 3.8s. The product owner requires sub-2s p90 latency and a fixed monthly serving budget. What is the most appropriate way to use these evaluation results to decide next steps?

Accepted Answer

Weigh the marginal 3% quality gain against the tripled token cost and latency SLA violation, and either revert or test cheaper alternatives (smaller model, tighter prompts, fewer retrieved chunks) that meet the latency and budget constraints.

Answer

Keep the larger model because any improvement in correctness and groundedness always outweighs cost and latency regressions in a production assistant.

Answer

Disable the cost and latency metrics in Agent Evaluation so the report focuses only on correctness and groundedness, which are the metrics that matter for quality.

Answer

Ignore the offline evaluation entirely and rely solely on production inference table monitoring, since offline token counts are not representative of real traffic.

Question 3

A GenAI engineer runs Mosaic AI Agent Evaluation on a customer-support RAG agent. The built-in judges (correctness, groundedness, relevance, safety) all pass, but the compliance team insists that every response must also avoid making promises about refund timelines, a domain-specific policy not captured by any built-in judge. The engineer wants this policy checked automatically on each evaluation run alongside the existing metrics. What is the most appropriate approach?

Accepted Answer

Define a custom LLM-judge metric with a guidelines/prompt describing the refund-timeline policy and pass it to mlflow.evaluate so it runs alongside the built-in judges.

Answer

Lower the safety judge's threshold so it also flags refund-timeline promises during evaluation.

Answer

Manually inspect every evaluation output for refund-timeline promises after each run and record findings in a spreadsheet.

Answer

Add the refund policy text to the system prompt and rely on the correctness judge to catch violations automatically.

Question 4

A Generative AI Engineer wants to run Mosaic AI Agent Evaluation on a deployed RAG chatbot using an offline evaluation dataset. The team has collected 200 representative user questions and expected reference answers, but no retrieval traces. They want to compute both answer correctness (comparing to the expected answers) and groundedness (whether responses are supported by retrieved context). Which approach correctly structures the evaluation so both metric types can be computed?

Accepted Answer

Provide a dataset with 'request' and 'expected_response' columns, and let Agent Evaluation invoke the agent so that retrieved context is captured in the trace for groundedness scoring.

Answer

Provide only a 'request' column with the 200 questions; Agent Evaluation will infer expected answers and retrieved context automatically.

Answer

Provide 'request' and 'expected_response' columns only; groundedness cannot be computed without manually pre-populating a 'retrieved_context' column for every row.

Answer

Split the work into two datasets—one with 'request' and 'expected_response' for correctness, and a separate one with only 'request' for groundedness—since the two metrics cannot share an evaluation run.

Question 5

A Generative AI Engineer is using Mosaic AI Agent Evaluation to assess a deployed RAG chatbot for a healthcare knowledge base. Stakeholders complain that some answers sound confident but contain claims not supported by the retrieved documents. The engineer has a golden evaluation dataset with questions but no ground-truth answers. Which built-in LLM-as-judge metric should the engineer prioritize to catch these unsupported claims?

Accepted Answer

Groundedness — measures whether the response is factually supported by the retrieved context

Answer

Answer correctness — compares the response against a labeled ground-truth answer

Answer

Chunk relevance — measures whether each retrieved chunk is relevant to the question

Answer

Latency — measures the end-to-end response time of the agent

Question 6

A Generative AI Engineer has built a RAG-based customer support agent and wants to systematically compare two candidate prompt templates before promoting either to production. They have assembled a curated dataset of representative questions, each with a reference (ground-truth) answer and the relevant supporting documents. They want to run Mosaic AI Agent Evaluation to measure answer correctness, groundedness, and retrieval relevance for both templates, then choose the better one. Which approach correctly uses Agent Evaluation for this offline comparison?

Accepted Answer

Run mlflow.evaluate() with the Agent Evaluation harness on the curated evaluation dataset for each template, comparing the built-in LLM-judge metrics (correctness, groundedness, retrieval relevance) across runs.

Answer

Deploy both templates to production serving endpoints first, then compare their inference table logs after a week of real traffic to decide which is better.

Answer

Enable Lakehouse Monitoring on the source Delta tables and use its data drift metrics to determine which template produces higher-quality answers.

Answer

Skip the labeled dataset and rely solely on measuring p95 latency and cost-per-request for each template, selecting whichever is cheaper and faster.

Question 7

A Generative AI Engineer runs Mosaic AI Agent Evaluation on a RAG chatbot. The aggregate scores show high groundedness but low answer relevance. The team wants to understand whether the problem originates in retrieval or in generation before making changes. Within the Agent Evaluation results, which artifact should they inspect FIRST to pinpoint the source of the low relevance?

Accepted Answer

The per-request detailed trace, which shows the retrieved chunks alongside the generated response and the individual judge assessments for each evaluated row

Answer

The overall aggregate metric summary table averaged across the entire evaluation dataset

Answer

The model serving endpoint's inference table payload logs from the last 24 hours of production traffic

Answer

The Lakehouse Monitoring drift dashboard comparing the current scoring window to the baseline table

Question 8

A Generative AI Engineer runs Mosaic AI Agent Evaluation on a deployed RAG chatbot. The overall answer-correctness score is low. The evaluation report shows that the retrieval metrics (context precision and context recall) are strong, but the answer-groundedness metric is poor. Which conclusion best explains this pattern, and what should the engineer prioritize fixing?

Accepted Answer

The correct context is being retrieved, but the LLM is generating claims not supported by that context; the engineer should focus on the generation prompt and model, not the retriever.

Answer

The retriever is pulling irrelevant documents; the engineer should tune the vector search index and re-embed the corpus.

Answer

The golden dataset labels are wrong; the engineer should discard the evaluation results and rely only on production inference tables.

Answer

The chunk size is too small; the engineer should increase chunk size to improve both retrieval and groundedness simultaneously.

Question 9

A GenAI engineer has a customer-support RAG agent deployed with inference tables enabled. Product leadership wants a weekly quality report showing, for each request, whether the response was grounded in retrieved context and whether it correctly answered the question. The team has thousands of daily requests and only two subject-matter experts who can review a small sample manually. Which approach best scales the per-request quality assessment while keeping human effort focused where it matters most?

Accepted Answer

Run Mosaic AI Agent Evaluation with LLM-as-judge metrics (e.g., correctness and groundedness) across all logged requests, then route only low-scoring or low-confidence cases to the two SMEs for manual review.

Answer

Have the two SMEs manually label every request each week, since only human judgment can reliably assess groundedness and correctness at the required quality.

Answer

Rely solely on Lakehouse Monitoring drift metrics on the inference table to infer whether response quality is acceptable, avoiding any per-request scoring.

Answer

Compute average response latency and token count per request from the inference table and use those as proxies for answer quality in the weekly report.

Question 10

A Generative AI Engineer has run Mosaic AI Agent Evaluation on version 1 of a RAG chatbot and logged the results as an MLflow run. After changing the retriever's chunking strategy and re-running the evaluation on the same evaluation dataset, the engineer wants to determine whether the change improved or degraded answer quality before promoting the new version. What is the most effective way to make this decision using the evaluation results?

Accepted Answer

Compare the aggregate LLM-as-judge and retrieval metrics across the two MLflow evaluation runs to see whether scores like groundedness and retrieval relevance improved or regressed.

Answer

Deploy both versions to production behind a traffic split and rely on user thumbs-up/thumbs-down feedback to decide which version is better.

Answer

Inspect the inference tables from the production endpoint for the new version to measure latency and token cost differences.

Answer

Manually re-read a random sample of 5 generated answers from the new version without referencing the version 1 results.

Question 11

A GenAI engineer deployed a RAG chatbot to a Model Serving endpoint two weeks ago. Leadership now wants a Lakehouse Monitoring dashboard that tracks request volume, latency trends, and response quality drift starting from launch day. When the engineer opens the endpoint, they discover no inference table exists and no payload logs were captured. What is the correct conclusion about building the requested historical dashboard?

Accepted Answer

Historical data from launch cannot be recovered; inference tables only capture payloads logged after the feature is enabled on the endpoint, so monitoring can only begin going forward once logging is turned on.

Answer

Lakehouse Monitoring can automatically backfill the inference table by replaying all historical requests stored in the endpoint's internal cache.

Answer

The engineer should enable MLflow autologging retroactively, which will reconstruct the two weeks of missing request and response payloads.

Answer

Because the endpoint already served traffic, the inference table exists by default and simply needs to be attached to a Lakehouse Monitoring profile.

Question 12

A Generative AI Engineer has deployed a RAG-based customer support assistant with inference tables enabled to capture request and response payloads. Over the past month, users have started asking about a newly launched product line that did not exist when the app was first evaluated. The engineer wants to detect that the distribution of incoming questions has shifted away from the topics covered in the original evaluation set, so they can decide when to refresh the knowledge base and re-run evaluation. Which approach best enables ongoing detection of this shift?

Accepted Answer

Attach a Lakehouse Monitoring inference profile to the inference table, define the original evaluation topic distribution as the baseline, and monitor for drift in the incoming request features.

Answer

Configure Lakehouse Monitoring on the inference table using a snapshot profile that only tracks row counts per day.

Answer

Increase the LLM-as-judge frequency to score every production response for correctness in real time.

Answer

Enable route-optimized serving on the endpoint to reduce latency for the new product questions.

Question 13

A GenAI engineer has deployed a customer-support RAG assistant with an inference table enabled and a Lakehouse Monitoring monitor of type 'InferenceLog' configured on top of it. The team wants the monitoring dashboards (drift, quality, and volume metrics) to reflect new production traffic automatically without an engineer manually triggering computation each morning. The monitor currently only updates when someone clicks 'Refresh metrics' in the UI. What is the most appropriate way to keep the monitor's metric tables and dashboard current on an ongoing basis?

Accepted Answer

Configure a schedule on the monitor so that its profile and drift metric tables are recomputed automatically at a defined cron interval.

Answer

Enable Delta Change Data Feed on the inference table so the monitor recomputes metrics on every row insert.

Answer

Increase the model serving endpoint's provisioned throughput so inference logs are written faster to the monitor tables.

Answer

Recreate the monitor as a 'Snapshot' type monitor, which continuously streams metrics as new data arrives.

Question 14

A GenAI engineer configures an LLM-as-judge metric in Mosaic AI Agent Evaluation to score answer correctness for a customer support RAG assistant. After the first evaluation run, stakeholders complain that the judge's scores don't match their intuition — several responses they consider clearly wrong were rated as correct. Before trusting the judge for ongoing quality gating, what is the most appropriate next step?

Accepted Answer

Assemble a small set of human-labeled examples and measure the LLM judge's agreement with those labels, then refine the judge's grading prompt/criteria until alignment is acceptable

Answer

Increase the judge model's temperature so it produces more varied scores that better reflect the range of answer quality

Answer

Switch the correctness metric to a purely lexical overlap score like BLEU, since LLM judges are inherently unreliable for correctness

Answer

Run the evaluation multiple times and average the scores to eliminate the disagreement with stakeholder intuition

Question 15

A Generative AI Engineer deploys a RAG chatbot on a Model Serving endpoint with inference tables enabled. Two weeks after launch, finance flags that the endpoint's monthly cost has tripled, even though the number of daily requests has stayed roughly flat. The engineer needs to determine the root cause before proposing a fix. Which analysis of the inference table data will most directly identify why costs increased despite stable request volume?

Accepted Answer

Aggregate the total input and output token counts per request over time and compare recent daily averages to the launch-week baseline

Answer

Compute the p95 request latency per day and check whether responses are being served more slowly than at launch

Answer

Measure the embedding drift of user queries against the launch-week baseline distribution using Lakehouse Monitoring

Answer

Count the number of distinct users querying the endpoint and correlate that with the billing period

Question 16

A Generative AI Engineer has deployed a customer-facing RAG chatbot on a Model Serving endpoint with payload logging enabled to an inference table. Product managers report that most users are satisfied with response speed, but a vocal minority complain about occasional multi-second delays. The engineer wants to quantify the experience of the worst-affected users and set an SLO alert. Which metric derived from the inference table timestamps should the engineer primarily track?

Accepted Answer

The p95 (95th percentile) request latency computed over a rolling time window

Answer

The mean (average) request latency across all logged requests

Answer

The total request count per hour

Answer

The median (p50) request latency across all logged requests

Question 17

A GenAI engineer deployed a customer-support RAG chatbot three months ago. Initial Mosaic AI Agent Evaluation runs on a golden dataset showed strong groundedness and correctness scores. Recently, users report increasingly stale or incorrect answers, though the chatbot code and prompts have not changed. The underlying knowledge-base Delta table (synced to the Vector Search index) is updated weekly by another team. The engineer wants an ongoing process to catch this kind of quality regression automatically rather than relying on user complaints. What is the most effective approach?

Accepted Answer

Schedule a recurring job that re-runs Mosaic AI Agent Evaluation against a maintained, periodically refreshed golden dataset and alerts when groundedness or correctness metrics drop below a defined threshold.

Answer

Enable Lakehouse Monitoring on the inference table and rely solely on request-volume and latency drift metrics to detect the quality regression.

Answer

Increase the retriever's top_k value so the chatbot always retrieves more chunks, reducing the chance of returning stale answers.

Answer

Re-log and re-deploy the chain weekly with pinned dependencies so the served model always matches the latest code version.

More Databricks GenAI Engineer Associate practice