Optimize generative AI systems and model performance

You are fine-tuning a GPT-4o model in Azure OpenAI to classify customer support tickets into 12 categories. Your training file contains 3,000 curated JSONL examples. After the first fine-tuning job, the training loss drops steadily but you have no way to tell whether the model will generalize to unseen tickets before you deploy it. You want to detect overfitting during the job itself and compare candidate hyperparameter configurations objectively. What should you do?

You are managing an Azure OpenAI fine-tuning job for a customer-support classification assistant. After the run completes, you review the training metrics and observe that training loss continues to decrease steadily across all 8 epochs, but validation loss stops improving after epoch 3 and begins to rise slightly. The deployed fine-tuned model performs noticeably worse on held-out real support tickets than a checkpoint you evaluated earlier. What is the most appropriate corrective action for the next fine-tuning run?

You fine-tune a GPT-4o model in Azure AI Foundry on a proprietary dataset of 800 curated examples. After training, the model performs excellently on the training set but produces rigid, low-quality responses on held-out validation prompts, and the training loss curve dropped rapidly while validation loss increased after the first epoch. You must adjust the fine-tuning hyperparameters to improve generalization. Which change is the most appropriate first step?

Your team must fine-tune a large open-source language model for a specialized legal-summarization task. You have a single GPU with limited VRAM and a modest labeled dataset. Full fine-tuning of all model weights repeatedly runs out of memory, and you also need to keep the fine-tuning artifact small so that multiple task-specific adapters can be swapped in and out on the same base model in production. Which fine-tuning approach best meets these requirements?

You are preparing a synthetic dataset to fine-tune a domain-specific chat model in Azure AI Foundry. The synthetic examples were generated by a larger teacher model using prompts derived from your internal knowledge base. Before training, you split the data into training and evaluation sets. After fine-tuning, the model reports excellent evaluation scores, but real-world performance on held-out user queries is noticeably worse. Which action should you take first to make the evaluation scores trustworthy?

You are fine-tuning a GPT-4o model for a legal contract-review assistant. Your production traffic contains many rare clause types (indemnification, force majeure, arbitration) that are underrepresented in your labeled training set of 800 examples. Human annotation is expensive and slow. You must expand and balance the training data before fine-tuning while minimizing the risk of degraded generalization. Which approach best addresses this?

You are fine-tuning a domain-specific model for a legal-tech company. To expand a small labeled dataset, you generate synthetic training examples using a larger teacher model. After the first fine-tuning run, evaluation shows the model reproduces factual errors and inconsistent formatting that were present in some synthetic examples. You must improve the fine-tuned model's quality before the next production candidate. What should you do FIRST with the synthetic data?

You fine-tuned a base language model on a large domain-specific dataset for a legal-document assistant. After deployment, evaluation shows the model now performs well on legal tasks but has significantly degraded on general reasoning and instruction-following tasks it previously handled. You must re-run the customization to preserve general capabilities while still gaining domain accuracy, using minimal compute. Which approach best addresses this problem?

Your team fine-tuned a GPT-4o model on 8,000 curated support-ticket examples to improve domain-specific responses. Before promoting it to production, you must decide whether the fine-tuned model actually outperforms the base model on your business task. You have a held-out test set with reference answers. Which approach provides the most defensible evidence to justify promotion?

You fine-tuned a GPT model in Azure AI Foundry for a customer-support workload. The model performed well on offline evaluation, but leadership requires evidence that it behaves correctly on real production traffic before it replaces the current model, without exposing customers to the new model's responses. Which promotion approach lets you validate the fine-tuned model against live traffic while ensuring users continue to receive responses only from the current production model?

You manage a fine-tuned GPT model deployed in Azure AI Foundry that summarizes legal contracts. Two months after deployment, users report that summaries have become less accurate, though no code or model changes were made. You suspect the input distribution has shifted as newer contract types are now being submitted. You need to confirm whether the fine-tuned model's output quality has degraded on the newer inputs before deciding to re-fine-tune. What should you do FIRST?

You maintain a production RAG assistant for an insurance company. After tuning chunk sizes and embedding models, you propose switching from pure semantic retrieval to hybrid (semantic + keyword) retrieval, expecting better answers on policy-number lookups without regressing general questions. Leadership wants statistically sound evidence before the change is promoted. Which approach best validates the improvement before full rollout?

Your team runs a RAG solution over a corpus of long technical manuals. During evaluation, users report that answers are frequently cut off mid-explanation and the retrieved passages often lack the surrounding context needed to answer procedural questions. Retrieval recall metrics show relevant documents are being found, but groundedness scores are low. You are currently using a fixed chunk size of 256 tokens with no overlap. Which change to your chunking strategy is MOST likely to improve answer completeness and groundedness?

Your production RAG system uses a general-purpose embedding model with 1,536 dimensions, and its vector index in Azure AI Search contains millions of chunked documents. Evaluation shows poor retrieval relevance for your specialized medical terminology. You fine-tune a domain-specific embedding model that outputs 768 dimensions and it scores significantly higher on your relevance benchmark. What must you do before the fine-tuned embedding model can serve production retrieval queries?

You operate a RAG solution for a legal firm. The knowledge base contains contracts full of specialized legal terminology and clause references. Users report that retrieval frequently returns generic passages instead of the precise clauses they ask about, even after you tuned chunk size and similarity threshold. Analysis shows the general-purpose embedding model places semantically distinct legal terms too close together in vector space. You want the most durable improvement to retrieval accuracy for this domain. What should you do?

You maintain a RAG solution over an internal engineering knowledge base in Azure AI Search. Users frequently query using exact part numbers and error codes (for example, "ERR-4821" and "P/N 9X-77"). Your current pure vector (semantic) retrieval often fails to return the document containing the exact identifier, even when it exists, because semantically similar chunks rank higher. You must improve retrieval accuracy for these exact-match queries without losing the conceptual matching that works well for natural-language questions. What should you do?

You maintain a RAG system for a legal research assistant. Users submit natural-language questions that frequently include specific statute citations (e.g., "Section 302(b)(2)"). Pure semantic vector search retrieves conceptually related passages but often misses documents containing the exact citation, while a keyword-only index returns the citation matches but misses paraphrased legal concepts. You have already enabled both a vector index and a keyword (BM25) index in Azure AI Search. What should you configure to maximize retrieval accuracy across both query types with a single ranked result set?

Your RAG solution for a legal research assistant retrieves the top 20 chunks from an Azure AI Search hybrid index, but users report that highly relevant passages often appear far down the passed context and the LLM misses them. Retrieval recall is high, but the final answers lack precision. You must improve which chunks the LLM actually receives without re-indexing or changing chunk sizes. What should you add to the retrieval pipeline?

You operate a RAG-based support assistant on Azure AI Foundry. Evaluation shows the assistant frequently misses relevant policy details that exist in the knowledge base (low answer completeness), yet groundedness scores remain acceptable. The retrieval pipeline currently returns the top 3 chunks per query using pure vector search. You need to improve completeness without significantly degrading groundedness or latency. Which change should you make FIRST?

Your team runs a RAG-based support assistant on Azure AI Foundry. Users complain that answers frequently include irrelevant passages that dilute accuracy, even though the correct passages are almost always retrieved. Offline evaluation confirms high retrieval recall but low precision, and many low-scoring chunks are being passed into the prompt context. You must reduce the inclusion of weakly related chunks while preserving the strong retrieval recall you already have. What is the most appropriate first tuning action?