Medium — Databricks GenAI Engineer Associate Practice Quiz

Question 1

A Generative AI Engineer is running Mosaic AI Agent Evaluation on a RAG-based customer support assistant. The evaluation dataset includes a column with expert-written reference answers for each question. The team wants to measure how factually and semantically close the agent's generated responses are to these reference answers, independent of whether the retrieved context supported them. Which built-in Agent Evaluation metric directly addresses this requirement?

Accepted Answer

answer correctness, which compares the agent's response against the provided ground-truth reference answers

Answer

groundedness, which checks whether the response is supported by the retrieved context

Answer

chunk relevance (retrieval precision), which measures how relevant retrieved documents are to the request

Answer

safety, which flags whether the response contains harmful or toxic content

Question 2

A GenAI engineer runs Mosaic AI Agent Evaluation on a customer-support RAG agent. The built-in judges (correctness, groundedness, relevance, safety) all pass, but the compliance team insists that every response must also avoid making promises about refund timelines, a domain-specific policy not captured by any built-in judge. The engineer wants this policy checked automatically on each evaluation run alongside the existing metrics. What is the most appropriate approach?

Accepted Answer

Define a custom LLM-judge metric with a guidelines/prompt describing the refund-timeline policy and pass it to mlflow.evaluate so it runs alongside the built-in judges.

Answer

Lower the safety judge's threshold so it also flags refund-timeline promises during evaluation.

Answer

Manually inspect every evaluation output for refund-timeline promises after each run and record findings in a spreadsheet.

Answer

Add the refund policy text to the system prompt and rely on the correctness judge to catch violations automatically.

Question 3

A Generative AI Engineer is using Mosaic AI Agent Evaluation to assess a deployed RAG chatbot for a healthcare knowledge base. Stakeholders complain that some answers sound confident but contain claims not supported by the retrieved documents. The engineer has a golden evaluation dataset with questions but no ground-truth answers. Which built-in LLM-as-judge metric should the engineer prioritize to catch these unsupported claims?

Accepted Answer

Groundedness — measures whether the response is factually supported by the retrieved context

Answer

Answer correctness — compares the response against a labeled ground-truth answer

Answer

Chunk relevance — measures whether each retrieved chunk is relevant to the question

Answer

Latency — measures the end-to-end response time of the agent

Question 4

A Generative AI Engineer has built a RAG-based customer support agent and wants to systematically compare two candidate prompt templates before promoting either to production. They have assembled a curated dataset of representative questions, each with a reference (ground-truth) answer and the relevant supporting documents. They want to run Mosaic AI Agent Evaluation to measure answer correctness, groundedness, and retrieval relevance for both templates, then choose the better one. Which approach correctly uses Agent Evaluation for this offline comparison?

Accepted Answer

Run mlflow.evaluate() with the Agent Evaluation harness on the curated evaluation dataset for each template, comparing the built-in LLM-judge metrics (correctness, groundedness, retrieval relevance) across runs.

Answer

Deploy both templates to production serving endpoints first, then compare their inference table logs after a week of real traffic to decide which is better.

Answer

Enable Lakehouse Monitoring on the source Delta tables and use its data drift metrics to determine which template produces higher-quality answers.

Answer

Skip the labeled dataset and rely solely on measuring p95 latency and cost-per-request for each template, selecting whichever is cheaper and faster.

Question 5

A Generative AI Engineer runs Mosaic AI Agent Evaluation on a RAG chatbot. The aggregate scores show high groundedness but low answer relevance. The team wants to understand whether the problem originates in retrieval or in generation before making changes. Within the Agent Evaluation results, which artifact should they inspect FIRST to pinpoint the source of the low relevance?

Accepted Answer

The per-request detailed trace, which shows the retrieved chunks alongside the generated response and the individual judge assessments for each evaluated row

Answer

The overall aggregate metric summary table averaged across the entire evaluation dataset

Answer

The model serving endpoint's inference table payload logs from the last 24 hours of production traffic

Answer

The Lakehouse Monitoring drift dashboard comparing the current scoring window to the baseline table

Question 6

A GenAI engineer has a customer-support RAG agent deployed with inference tables enabled. Product leadership wants a weekly quality report showing, for each request, whether the response was grounded in retrieved context and whether it correctly answered the question. The team has thousands of daily requests and only two subject-matter experts who can review a small sample manually. Which approach best scales the per-request quality assessment while keeping human effort focused where it matters most?

Accepted Answer

Run Mosaic AI Agent Evaluation with LLM-as-judge metrics (e.g., correctness and groundedness) across all logged requests, then route only low-scoring or low-confidence cases to the two SMEs for manual review.

Answer

Have the two SMEs manually label every request each week, since only human judgment can reliably assess groundedness and correctness at the required quality.

Answer

Rely solely on Lakehouse Monitoring drift metrics on the inference table to infer whether response quality is acceptable, avoiding any per-request scoring.

Answer

Compute average response latency and token count per request from the inference table and use those as proxies for answer quality in the weekly report.

Question 7

You are building a Mosaic AI agent on Databricks that answers customer questions by calling multiple tools (a product catalog lookup, an inventory checker, and a shipping estimator). During testing, some queries cause the agent to enter long reasoning loops, repeatedly invoking tools without converging on a final answer. This drives up latency and token costs. Which configuration change most directly prevents these runaway loops while still allowing legitimate multi-step reasoning?

Accepted Answer

Set a max_iterations (maximum reasoning steps) limit on the agent executor so it terminates and returns after a bounded number of tool-calling cycles.

Answer

Switch the underlying foundation model from a pay-per-token endpoint to a provisioned throughput endpoint.

Answer

Increase the model's max_tokens output parameter so it can produce a complete answer in a single response.

Answer

Reduce the temperature parameter to 0 so the model produces fully deterministic tool calls.

Question 8

You are building a Mosaic AI agent that has access to three registered Unity Catalog functions: `get_order_status`, `lookup_product_inventory`, and `search_return_policy`. During testing, you notice the LLM frequently calls `search_return_policy` even when users ask about where their package is. The tool bindings and endpoint work correctly, and each function executes without error when called directly. What is the most effective change to improve the agent's tool selection accuracy?

Accepted Answer

Write clear, distinct docstrings and parameter descriptions for each function so the LLM can accurately match user intent to the correct tool

Answer

Reduce the temperature of the foundation model endpoint to 0 to make tool calls deterministic

Answer

Increase the agent's maximum iteration limit so it has more chances to call the right tool

Answer

Switch the endpoint from a pay-per-token model to provisioned throughput for more consistent responses

Question 9

You are building a Mosaic AI agent that calls an external inventory API tool to answer customer stock-availability questions. During testing, the API intermittently returns timeouts and malformed responses, causing the agent to either crash or return the raw error text to the user. You need to make the agent robust while keeping responses helpful. Which approach best addresses this within the agent's tool-execution logic?

Accepted Answer

Wrap the tool call with retry-with-backoff logic and a fallback that returns a structured, user-friendly message the LLM can incorporate when the tool ultimately fails

Answer

Increase the foundation model's max_tokens so the LLM can absorb and rewrite the raw API error text before responding

Answer

Remove the inventory tool from the agent and instead embed all stock data into the system prompt so no external call is needed

Answer

Lower the model temperature to 0 so the agent stops producing inconsistent responses when the API fails

Question 10

You are building a Mosaic AI agent that calls a Unity Catalog function to look up a customer's order by an order ID. During testing, the LLM occasionally passes malformed arguments — sometimes an empty string, sometimes a full sentence like 'the order from yesterday' — causing the tool to fail. You want to reduce these failures by making the tool's expected input explicit to the LLM so it produces correctly formatted arguments more reliably. Which approach most directly addresses this?

Accepted Answer

Define a strict argument schema (e.g., a Pydantic model or typed function signature with descriptions) for the tool so the LLM receives a clear structured specification of the required input format.

Answer

Increase the model temperature so the agent explores a wider variety of argument formats until one succeeds.

Answer

Wrap the tool in a retry loop that re-invokes it with the same arguments until the call returns successfully.

Answer

Raise the agent's maximum iteration limit so it has more attempts to eventually call the tool correctly.

Question 11

You are building a customer-facing shopping assistant on Databricks. Customers ask two kinds of questions: (1) explanations of the store's return and warranty policies, which are documented in a static PDF knowledge base, and (2) current stock levels and prices for specific SKUs, which change minute-to-minute and live in a transactional database exposed through a REST API. You want a single Mosaic AI agent that answers both accurately. Which design correctly handles both question types?

Accepted Answer

Use a RAG retriever over the policy documents for policy questions, and register a tool (function) that calls the live inventory REST API for stock and price questions, letting the agent choose based on the query.

Answer

Index both the policy PDFs and a nightly snapshot of the inventory table in Vector Search, and answer every question with pure RAG retrieval.

Answer

Fine-tune a foundation model on both the policy PDFs and current inventory data so it can answer all questions from its weights.

Answer

Store the inventory data as few-shot examples in the system prompt and use RAG only for the policy documents.

Question 12

A customer support team receives tickets in multiple languages. The GenAI application must: (1) detect and translate each ticket into English, (2) classify the ticket into a support category, and (3) generate a concise English summary for the support agent. The team wants a maintainable design where each step can be tested and swapped independently, and where a failure or low-confidence result in one step can be inspected before the next step runs. Which design approach best meets these requirements?

Accepted Answer

Decompose the workflow into a sequential chain of discrete steps (translate → classify → summarize), where the output of each step is passed as input to the next and each step is an independently testable component.

Answer

Build a single large prompt that instructs one LLM call to translate, classify, and summarize the ticket all at once, returning a combined result.

Answer

Fine-tune a single model on translated, classified, and summarized examples so the model performs all three tasks implicitly without separate steps.

Answer

Deploy three completely independent applications that each process the raw ticket in parallel, then merge their unrelated outputs at the end.

Question 13

You are building a multi-turn customer support agent on Databricks that uses a foundation model with an 8,192-token context window. Each turn appends the full conversation history plus a system prompt plus retrieved documents. After roughly 15 exchanges, users start receiving errors that the context length has been exceeded. The business requires that the agent still remember key facts from earlier in the conversation (like the customer's account ID and reported issue). What is the most appropriate way to manage context so the agent keeps working across long conversations?

Accepted Answer

Use a conversation summary memory that progressively condenses older turns into a running summary while keeping the most recent turns verbatim.

Answer

Increase the max_tokens generation parameter so the model has more room to produce output.

Answer

Truncate the conversation by dropping the oldest turns entirely once the context limit is approached.

Answer

Switch to a larger provisioned-throughput endpoint of the same model to raise the throughput limit.

Question 14

A Generative AI Engineer is preparing a knowledge base of long-form product manuals for a RAG chatbot. During evaluation, users report that answers frequently seem 'cut off' or miss information that spans across paragraph boundaries. The team is using fixed-size chunking of 500 tokens with zero overlap. Which adjustment to the chunking strategy is MOST likely to resolve this issue while keeping chunk sizes manageable?

Accepted Answer

Introduce a chunk overlap (e.g., 50-100 tokens) between consecutive chunks so context that spans boundaries is preserved in adjacent chunks.

Answer

Increase the fixed chunk size to 4000 tokens with zero overlap so each chunk contains more complete information.

Answer

Switch to character-based chunking of exactly 500 characters with zero overlap to make chunks more granular.

Answer

Remove chunking entirely and embed each full manual as a single vector to avoid losing any information.

Question 15

A Generative AI Engineer is preparing a corpus of long-form product manuals (avg 40 pages each) for a RAG application. They plan to use an embedding model with a maximum input context of 512 tokens. During testing, they notice that chunks frequently exceed this limit and are silently truncated, causing the tail of each chunk's content to be excluded from the embedding. What is the most appropriate action to ensure complete content is represented in the vector index?

Accepted Answer

Reduce the target chunk size so that each chunk (plus any overlap) stays within the embedding model's 512-token limit before generating embeddings.

Answer

Keep the large chunks but increase the number of retrieved results (top-k) at query time to compensate for truncated content.

Answer

Concatenate multiple manuals into a single document before chunking so the embedding model sees more context.

Answer

Switch to a larger vector dimension in the index so more of each chunk's tokens can be stored.

Question 16

You are building a RAG application over a customer support knowledge base composed of thousands of short FAQ entries. Each entry is a self-contained question paired with a concise answer, and entries are typically 2-4 sentences long. Retrieval accuracy is currently poor because chunks frequently split a question from its answer or merge multiple unrelated FAQs together. Which chunking strategy best fixes this?

Accepted Answer

Treat each FAQ question-and-answer pair as a single chunk, aligning chunk boundaries to the natural document structure.

Answer

Use a fixed-size character chunker of 512 characters with 50-character overlap applied uniformly across the concatenated document.

Answer

Chunk at the sentence level so each individual sentence becomes its own embedding for maximum granularity.

Answer

Merge the entire knowledge base into one large chunk so all context is always available to the retriever.

Question 17

You are preparing a large internal knowledge base for a RAG application. The documents are Markdown files with clear hierarchical headers (H1 for product areas, H2 for features, H3 for individual procedures). Users typically ask specific procedural questions like 'How do I reset the API key for the billing service?'. You want each retrieved chunk to be self-contained and topically coherent so the LLM has focused context. Which chunking approach best fits these documents?

Accepted Answer

Use header-aware (structure-based) splitting that segments at Markdown headers and prepends the parent header path to each chunk

Answer

Split the text into fixed 512-token windows with no overlap, ignoring the Markdown structure

Answer

Store each entire Markdown file as a single chunk to preserve all context

Answer

Split on every sentence boundary to maximize the number of retrievable units

Question 18

You are building a RAG system over a corpus of customer support tickets. The tickets vary widely in length: many are short one-paragraph questions, but some are long multi-issue threads spanning several distinct problems. Users query for solutions to specific individual problems. You want each retrieved chunk to represent a coherent, self-contained unit so that retrieval returns focused answers without mixing unrelated issues. Which chunking approach best fits this requirement?

Accepted Answer

Use semantic (content-aware) chunking that splits threads at topic boundaries so each chunk captures one distinct issue

Answer

Apply a fixed-size token window (e.g., 512 tokens) uniformly across all tickets regardless of structure

Answer

Store each entire ticket as a single chunk regardless of length to preserve full context

Answer

Split every ticket into single-sentence chunks to maximize retrieval granularity

Question 19

You are building a RAG pipeline over quarterly financial reports delivered as PDFs. Each report contains long narrative sections interspersed with large tabular data (balance sheets, income statements). During testing, users ask questions about specific line-item values, but retrieved chunks frequently return partial tables where column headers are separated from their data rows, producing wrong or unanswerable results. Which document preparation approach best addresses this problem?

Accepted Answer

Use a table-aware extraction step that detects and serializes each table as a self-contained unit (headers plus rows) into its own chunk, while chunking narrative text separately with recursive character splitting

Answer

Increase the fixed chunk size to 4000 tokens so entire tables are more likely to fit inside a single chunk

Answer

Lower the embedding dimension to capture tabular structure more compactly and improve retrieval recall

Answer

Strip all tables from the PDFs during ingestion and rely solely on the narrative sections for answering questions

Question 20

You are building a RAG application over a large corpus of software engineering documentation that mixes prose explanations with fenced code blocks and API tables. During testing, users complain that when they ask 'show me the function that validates tokens,' the retrieved chunks return only the surrounding explanation text and cut code samples in half, making the answers useless. You are currently using a fixed-size character chunker with 500-character chunks and no overlap. Which change to your chunking strategy will most directly improve retrieval of complete, coherent code examples?

Accepted Answer

Switch to a structure-aware (recursive/markdown-aware) chunker that respects code-fence and section boundaries, and add chunk overlap to preserve context.

Answer

Reduce the chunk size to 200 characters so more granular chunks are indexed and code is easier to match.

Answer

Increase the number of results (top_k) returned by Vector Search from 3 to 50 so the code block is eventually included.

Answer

Replace the embedding model with a larger general-purpose text embedding model to better understand code semantics.

Question 21

A Generative AI Engineer has run Mosaic AI Agent Evaluation on version 1 of a RAG chatbot and logged the results as an MLflow run. After changing the retriever's chunking strategy and re-running the evaluation on the same evaluation dataset, the engineer wants to determine whether the change improved or degraded answer quality before promoting the new version. What is the most effective way to make this decision using the evaluation results?

Accepted Answer

Compare the aggregate LLM-as-judge and retrieval metrics across the two MLflow evaluation runs to see whether scores like groundedness and retrieval relevance improved or regressed.

Answer

Deploy both versions to production behind a traffic split and rely on user thumbs-up/thumbs-down feedback to decide which version is better.

Answer

Inspect the inference tables from the production endpoint for the new version to measure latency and token cost differences.

Answer

Manually re-read a random sample of 5 generated answers from the new version without referencing the version 1 results.

Question 22

A software company wants to build a generative AI assistant that reviews pull requests. For each PR, the assistant must: (1) summarize the code changes, (2) flag potential security vulnerabilities against the company's internal secure-coding guidelines, and (3) generate a natural-language review comment that references specific findings. The security guidelines are a large, frequently updated internal document. As the GenAI engineer designing this application, what is the most appropriate high-level architecture?

Accepted Answer

A chain that first summarizes the diff, then performs a RAG step retrieving relevant guideline sections from a vector index to identify vulnerabilities, then composes the review comment from the prior outputs

Answer

A single prompt that pastes the full secure-coding guidelines and the entire code diff into one LLM call, asking it to return the summary, vulnerabilities, and review comment together

Answer

Fine-tune a base model on the secure-coding guidelines and use it directly to produce all three outputs from the raw diff in one step

Answer

Three fully independent LLM calls that each receive only the raw diff, with no shared context or retrieval, and concatenate their outputs

Question 23

A financial services company wants to build a customer support assistant that can answer questions about a customer's account. A single request may require the assistant to: (1) look up the customer's current account balance from an internal transactional database, (2) retrieve relevant policy explanations from an internal knowledge base of PDFs, and (3) perform a currency conversion calculation before presenting the final answer. The team wants a design where the LLM dynamically decides which capability to invoke based on the user's question. Which application design best fits these requirements?

Accepted Answer

An agent that reasons over the request and selects among defined tools: a SQL/function-calling tool for the balance lookup, a vector-search retriever tool for policy documents, and a calculator/code tool for currency conversion

Answer

A single large prompt that concatenates the full account database, all policy PDFs, and conversion tables into the context window for every request

Answer

A pure RAG pipeline that embeds the user question, retrieves the top-k policy chunks, and passes them to the LLM to generate all answers including balances and conversions

Answer

A fine-tuned model trained on historical account balances and policy text so it can answer all three types of questions without any external calls

Question 24

A Generative AI Engineer is building a RAG application over a corpus of internal support articles that were exported from three different systems. During data profiling, they discover that roughly 30% of the articles are near-duplicates (same content with minor formatting differences) because articles were migrated multiple times. Early retrieval tests show that the top-k results are frequently filled with several copies of the same article, crowding out other relevant content. What is the most effective step to improve retrieval quality before generating embeddings?

Accepted Answer

Deduplicate the source documents during preprocessing (e.g., using content hashing or similarity-based clustering) so only one canonical version of each article is embedded and indexed

Answer

Increase the number of retrieved documents (k) so that non-duplicate content is more likely to appear in the results

Answer

Switch to a higher-dimensional embedding model so duplicate articles produce more distinct vectors

Answer

Reduce the chunk size so that duplicate articles are split into smaller, less-overlapping segments

Question 25

You are designing a customer-support generative AI application for an e-commerce company. Business requirements state that the assistant must: (1) answer general product policy questions, (2) look up a specific customer's live order status by order ID, and (3) initiate refund requests that require writing to an internal transactional system. The company wants a single conversational entry point that decides which capability to use per turn. Which application design best satisfies these requirements?

Accepted Answer

An agent that uses an LLM to reason about intent and select among tools: a RAG retriever over policy documents, a read-only order-lookup function, and a refund-submission function that writes to the transactional system.

Answer

A single large prompt that concatenates the policy documents, the order database schema, and refund instructions, relying on the LLM to generate answers, order statuses, and refund confirmations directly from context.

Answer

A fixed sequential chain that always retrieves policy documents, then queries order status, then submits a refund on every user turn regardless of the question asked.

Answer

A fine-tuned model trained on historical support transcripts, deployed without external tools, that memorizes policies, order data, and refund outcomes.