Question

You are preparing a synthetic dataset to fine-tune a domain-specific chat model in Azure AI Foundry. The synthetic examples were generated by a larger teacher model using prompts derived from your internal knowledge base. Before training, you split the data into training and evaluation sets. After fine-tuning, the model reports excellent evaluation scores, but real-world performance on held-out user queries is noticeably worse. Which action should you take first to make the evaluation scores trustworthy?

Accepted Answer

Deduplicate near-identical synthetic examples across the training and evaluation splits to prevent data leakage between them

Answer

Increase the learning rate multiplier so the model generalizes better to unseen queries

Answer

Add more synthetic examples generated from the same prompts to expand the training set

Answer

Switch the evaluation metric from exact match to a coherence metric