Question

Your team fine-tuned a GPT-4o model on 8,000 curated support-ticket examples to improve domain-specific responses. Before promoting it to production, you must decide whether the fine-tuned model actually outperforms the base model on your business task. You have a held-out test set with reference answers. Which approach provides the most defensible evidence to justify promotion?

Accepted Answer

Evaluate both the base model and the fine-tuned model on the same held-out test set using task-relevant metrics, then compare the aggregated scores side by side.

Answer

Run the fine-tuning job with a higher number of epochs and confirm that the training loss curve decreases smoothly across all epochs.

Answer

Compare the token cost per request of the fine-tuned deployment against the base model deployment and promote whichever is cheaper.

Answer

Manually review 10 responses from the fine-tuned model and promote it if they read well to a subject-matter expert.