Hard — Databricks GenAI Engineer Associate Practice Quiz

Question 1

A team runs Mosaic AI Agent Evaluation nightly on a golden dataset of 500 questions for their production RAG assistant. After a recent change that swapped the generation model to a larger foundation model, the evaluation report shows the average correctness and groundedness scores each improved by about 3%, but the total_input_token_count and total_output_token_count metrics roughly tripled, and mean response latency rose from 1.2s to 3.8s. The product owner requires sub-2s p90 latency and a fixed monthly serving budget. What is the most appropriate way to use these evaluation results to decide next steps?

Accepted Answer

Weigh the marginal 3% quality gain against the tripled token cost and latency SLA violation, and either revert or test cheaper alternatives (smaller model, tighter prompts, fewer retrieved chunks) that meet the latency and budget constraints.

Answer

Keep the larger model because any improvement in correctness and groundedness always outweighs cost and latency regressions in a production assistant.

Answer

Disable the cost and latency metrics in Agent Evaluation so the report focuses only on correctness and groundedness, which are the metrics that matter for quality.

Answer

Ignore the offline evaluation entirely and rely solely on production inference table monitoring, since offline token counts are not representative of real traffic.

Question 2

A Generative AI Engineer wants to run Mosaic AI Agent Evaluation on a deployed RAG chatbot using an offline evaluation dataset. The team has collected 200 representative user questions and expected reference answers, but no retrieval traces. They want to compute both answer correctness (comparing to the expected answers) and groundedness (whether responses are supported by retrieved context). Which approach correctly structures the evaluation so both metric types can be computed?

Accepted Answer

Provide a dataset with 'request' and 'expected_response' columns, and let Agent Evaluation invoke the agent so that retrieved context is captured in the trace for groundedness scoring.

Answer

Provide only a 'request' column with the 200 questions; Agent Evaluation will infer expected answers and retrieved context automatically.

Answer

Provide 'request' and 'expected_response' columns only; groundedness cannot be computed without manually pre-populating a 'retrieved_context' column for every row.

Answer

Split the work into two datasets—one with 'request' and 'expected_response' for correctness, and a separate one with only 'request' for groundedness—since the two metrics cannot share an evaluation run.

Question 3

A Generative AI Engineer runs Mosaic AI Agent Evaluation on a deployed RAG chatbot. The overall answer-correctness score is low. The evaluation report shows that the retrieval metrics (context precision and context recall) are strong, but the answer-groundedness metric is poor. Which conclusion best explains this pattern, and what should the engineer prioritize fixing?

Accepted Answer

The correct context is being retrieved, but the LLM is generating claims not supported by that context; the engineer should focus on the generation prompt and model, not the retriever.

Answer

The retriever is pulling irrelevant documents; the engineer should tune the vector search index and re-embed the corpus.

Answer

The golden dataset labels are wrong; the engineer should discard the evaluation results and rely only on production inference tables.

Answer

The chunk size is too small; the engineer should increase chunk size to improve both retrieval and groundedness simultaneously.

Question 4

You are building a multi-step agent that answers customer questions about their orders. The workflow requires the agent to (1) look up a customer's order ID from their email, (2) retrieve the shipment status using that order ID, and (3) generate a natural-language summary. During testing, the agent frequently fails at step 2 because it invokes the shipment-status tool with the customer's email instead of the order ID returned by step 1. What is the most effective way to fix this behavior?

Accepted Answer

Ensure each tool returns clearly structured output (e.g., a JSON field named order_id) and write tool descriptions that explicitly state which input each tool requires, so the agent can carry the intermediate result forward

Answer

Increase the agent's max_iterations so it has more attempts to eventually call the shipment-status tool with the correct argument

Answer

Combine all three steps into a single tool so the agent never has to pass intermediate results between calls

Answer

Switch the underlying foundation model to a larger pay-per-token model with a bigger context window

Question 5

You are building a Mosaic AI agent for a retail customer-support app. The agent has access to a tool `issue_refund(order_id, amount)` that calls an internal API to process a monetary refund. During testing, you notice that when the LLM's reasoning loop occasionally re-invokes the same tool call after a transient response parsing error, customers are sometimes refunded twice for the same order. Which design change most directly prevents duplicate side effects from repeated tool invocations?

Accepted Answer

Add an idempotency key (e.g., a hash of order_id + refund request ID) so the refund API treats repeated calls for the same order as a single operation

Answer

Increase the agent's max_iterations so it has more attempts to complete the reasoning loop successfully

Answer

Lower the model temperature to 0 so the agent produces more deterministic reasoning and never repeats a tool call

Answer

Rewrite the tool description to instruct the LLM to call issue_refund only once per conversation

Question 6

You are building a Mosaic AI agent that answers customer queries like 'Compare the shipping cost and estimated delivery date for my order to the standard rate.' The agent has three tools: get_order_details (returns order ID metadata), get_shipping_cost (requires the order's weight and destination from get_order_details), and get_standard_rate (returns a fixed published rate with no inputs). You want to minimize end-to-end latency without producing incorrect tool inputs. How should you structure the agent's tool execution?

Accepted Answer

Run get_order_details first, then run get_shipping_cost and get_standard_rate in parallel once its output is available.

Answer

Run all three tools in parallel at the start to minimize latency, then reconcile any missing inputs afterward.

Answer

Run all three tools strictly sequentially in the order they were registered to guarantee correct inputs.

Answer

Merge get_order_details and get_shipping_cost into a single tool and skip get_standard_rate since rates are published.

Question 7

A financial services firm wants to build a system that answers analyst questions like 'Summarize the risk factors from the three latest 10-K filings for Company X and compare them to sector peers.' The number of documents to retrieve and the comparison steps vary per query, and the system sometimes needs to call a live market-data API mid-workflow depending on what the question asks. The team must decide how to structure the application. Which design approach best fits these requirements?

Accepted Answer

An agent with access to a retrieval tool and a market-data API tool, allowing the LLM to reason about which tools to invoke and in what order per query

Answer

A fixed sequential chain with predefined steps: retrieve documents, summarize, then compare, using the same tool order for every query

Answer

A single large prompt that stuffs all possible 10-K filings and market data into the context window and asks the model to answer directly

Answer

A batch fine-tuning job that trains the base model on all historical 10-K filings so it can answer from parametric memory

Question 8

A Generative AI Engineer is designing a customer-support RAG assistant for a mid-size retailer. Business requirements state that roughly 80% of incoming questions are simple FAQ-style lookups (store hours, return policy) that must be answered in under 1 second at low cost, while the remaining 20% are complex, multi-part troubleshooting questions where answer quality matters most and slightly higher latency is acceptable. The team has both a small, fast, inexpensive LLM and a large, high-quality, more expensive LLM available as Model Serving endpoints. Which application design best satisfies these requirements?

Accepted Answer

Add a lightweight query-classification/routing step that sends simple FAQ queries to the small fast model and routes complex troubleshooting queries to the large high-quality model.

Answer

Route every query to the large high-quality model to guarantee the best possible answer quality across all question types.

Answer

Route every query to the small fast model and rely on more retrieved context chunks to compensate for its lower reasoning ability on complex questions.

Answer

Fine-tune the small fast model on the complete FAQ and troubleshooting corpus so a single model can handle both query types without any routing.

Question 9

A GenAI engineer built a RAG chain in a Databricks notebook using LangChain. The chain references a custom Python helper module (retrieval_utils.py) that lives alongside the notebook. When logging the chain with MLflow using mlflow.langchain.log_model() by passing the in-memory chain object directly, the model deploys successfully but fails at serving time with a ModuleNotFoundError for retrieval_utils. What is the recommended way to log the model so it serves correctly?

Accepted Answer

Use MLflow's code-based (Models from Code) logging by passing the path to a Python script that defines and sets the chain, and include the helper module via code_paths so dependencies are captured.

Answer

Increase the serving endpoint compute size so it has enough memory to import the helper module at runtime.

Answer

Manually copy retrieval_utils.py into the Databricks workspace root directory after the endpoint is created.

Answer

Set scale-to-zero to false so the endpoint keeps the notebook session warm and can access the helper module.

Question 10

A Generative AI Engineer builds a RAG chain locally using LangChain 0.2.x and a specific version of the databricks-vectorsearch SDK. After logging the chain with MLflow and deploying it to a Model Serving endpoint, the endpoint fails to start with import errors that never appeared during local development. The engineer confirms the code is identical. What is the most likely cause and correct fix?

Accepted Answer

The serving environment rebuilt with different library versions than local; explicitly capture and pin dependencies (e.g., via the model's inferred pip_requirements or an extra_pip_requirements list) when logging the model so the serving container matches the development environment.

Answer

Model Serving does not support LangChain chains; the engineer must rewrite the chain as a pyfunc model with no external dependencies before deploying.

Answer

The endpoint needs scale-to-zero disabled so the container stays warm and can resolve imports; enable provisioned concurrency to fix the import errors.

Answer

The Unity Catalog model registration stripped the dependency metadata; re-register the model using the Workspace Model Registry instead of Unity Catalog to preserve libraries.

Question 11

A Generative AI Engineer has built a RAG chain that queries a Databricks Vector Search index and calls a Foundation Model API endpoint. When they log the chain with MLflow using the models-from-code approach and deploy it to a Model Serving endpoint, the endpoint starts but every inference request fails with authentication errors when the chain tries to reach the Vector Search index and the LLM endpoint. What should the engineer do to fix this before re-logging the model?

Accepted Answer

Declare the Vector Search index and the serving endpoint in the 'resources' parameter of mlflow.langchain.log_model so Databricks provisions short-lived credentials for those dependencies at serving time.

Answer

Hardcode a personal access token into the chain's Python source file so the deployed model can authenticate to both dependencies.

Answer

Increase the endpoint's compute size and enable scale-to-zero so the model has enough resources to establish authenticated connections.

Answer

Set the workspace default catalog and schema in the MLflow experiment tags so the serving endpoint inherits the correct permissions.

Question 12

A Generative AI Engineer is logging a RAG chain with MLflow before deploying it to a Model Serving endpoint. They pass an input_example when calling mlflow.langchain.log_model() but do not explicitly define a ModelSignature. When they later query the deployed endpoint, some client requests are rejected due to schema mismatch, while the engineer expected MLflow to have captured a signature automatically. What is the most accurate explanation of MLflow's behavior here and how to make the endpoint's input contract reliable?

Accepted Answer

MLflow can infer a signature from the input_example, but inference from a single example may not capture all valid field types or optional keys; explicitly defining and logging a ModelSignature makes the input/output contract deterministic for serving.

Answer

MLflow never infers a signature from an input_example, so no signature was stored at all; the only fix is to disable schema enforcement on the serving endpoint.

Answer

Providing an input_example automatically guarantees a complete and correct signature for all future requests, so the mismatch must be caused by the endpoint's autoscaling configuration rather than the schema.

Answer

Signatures are only supported for pyfunc models, not for the langchain flavor, so the engineer must re-log the chain as a custom pyfunc model to get any schema enforcement.

Question 13

A GenAI engineer deploys a custom PyFunc RAG chain to a Databricks Model Serving endpoint. During a product launch, the endpoint receives sustained traffic of roughly 200 concurrent requests, but users report frequent HTTP 429 (Too Many Requests) errors even though CPU utilization on the compute is low. The endpoint is configured with a fixed compute size and a maximum concurrency well below the incoming load. What is the most appropriate action to eliminate the 429 errors while handling the sustained traffic?

Accepted Answer

Increase the endpoint's concurrency by scaling up the provisioned compute (larger workload size and/or higher max provisioned concurrency) so it can serve the sustained request volume.

Answer

Enable scale-to-zero on the endpoint so idle capacity is reclaimed and reallocated to active requests.

Answer

Switch the endpoint from CPU to GPU compute so each request completes faster and reduces queueing.

Answer

Reduce the retriever's top_k parameter in the chain to lower the payload size returned to clients.

Question 14

A GenAI engineer deploys a fine-tuned 13B-parameter LLM to a Databricks Model Serving endpoint using a GPU compute type. The endpoint deployment repeatedly fails with a container initialization timeout during the model loading phase, even though the same model loads successfully in an interactive notebook on a comparable GPU cluster. The MLflow model was logged with the correct signature and all pip dependencies pinned. What is the MOST likely cause and appropriate remediation?

Accepted Answer

The model artifact and its large weight files exceed what can be loaded within the endpoint's startup readiness window; increase the workload size to a larger GPU tier so weights load faster and the container passes its readiness check in time.

Answer

The MLflow signature is invalid for GPU serving; re-log the model without a signature so Model Serving can infer the schema dynamically at load time.

Answer

Scale-to-zero is enabled, which prevents GPU endpoints from initializing; disable scale-to-zero and the container will initialize immediately.

Answer

The pinned dependencies conflict with the endpoint runtime; remove all version pins so the serving environment installs the latest compatible packages automatically.

Question 15

A GenAI engineer has deployed a custom RAG chain to a Databricks Model Serving endpoint that fronts a customer-facing chatbot. During load testing, the team observes that at high request volumes the endpoint adds noticeable per-request latency from the serving proxy layer, and they need to sustain thousands of concurrent queries with the lowest possible routing overhead. Which endpoint configuration change should the engineer make to address this?

Accepted Answer

Enable route optimization on the serving endpoint to reduce network and proxy overhead for high-throughput, low-latency traffic

Answer

Enable scale-to-zero so idle compute is released and reallocated to active requests

Answer

Switch the endpoint from provisioned concurrency to pay-per-token billing to remove routing limits

Answer

Increase the chunk overlap in the RAG retriever so fewer retrieval calls are needed per request

Question 16

A GenAI engineer deployed a customer-support RAG chatbot three months ago. Initial Mosaic AI Agent Evaluation runs on a golden dataset showed strong groundedness and correctness scores. Recently, users report increasingly stale or incorrect answers, though the chatbot code and prompts have not changed. The underlying knowledge-base Delta table (synced to the Vector Search index) is updated weekly by another team. The engineer wants an ongoing process to catch this kind of quality regression automatically rather than relying on user complaints. What is the most effective approach?

Accepted Answer

Schedule a recurring job that re-runs Mosaic AI Agent Evaluation against a maintained, periodically refreshed golden dataset and alerts when groundedness or correctness metrics drop below a defined threshold.

Answer

Enable Lakehouse Monitoring on the inference table and rely solely on request-volume and latency drift metrics to detect the quality regression.

Answer

Increase the retriever's top_k value so the chatbot always retrieves more chunks, reducing the chance of returning stale answers.

Answer

Re-log and re-deploy the chain weekly with pinned dependencies so the served model always matches the latest code version.

Question 17

A GenAI engineer deploys a RAG chatbot as a Model Serving endpoint. The endpoint queries a Vector Search index and reads a Delta table that contains sales records restricted by row-level filters. The compliance team requires that each end user only sees rows they are individually entitled to, rather than every user seeing the same data through the service principal that owns the endpoint. Which approach best satisfies this requirement while using Unity Catalog governance?

Accepted Answer

Enable on-behalf-of-user authentication (user-scoped credentials) for the serving endpoint so Unity Catalog evaluates row-level and column-level policies against the invoking user's identity

Answer

Grant the endpoint's service principal full SELECT on the Delta table and filter rows in the application code based on the username passed in the request payload

Answer

Create a separate serving endpoint per user and grant each endpoint's service principal only the rows that user is allowed to see

Answer

Disable row-level security on the source table and rely on the vector index metadata filters to enforce per-user entitlements