Question

A media company wants to select between two candidate foundation models for a customer-facing product-description generator. They have assembled a fixed set of 500 product inputs with high-quality reference descriptions written by their editorial team. Before launch, they want an objective, repeatable way to compare which model produces output closest to their reference standard across the full test set. Which evaluation approach best meets this need?

Accepted Answer

Run both models against the 500-input benchmark dataset and compare automated similarity metrics (such as BLEU or ROUGE) against the reference descriptions

Answer

Deploy both models to production and monitor which one generates fewer customer support tickets over the first month

Answer

Ask each model to rate its own output quality on a scale of 1 to 10 and select the model with the higher average self-reported score

Answer

Increase the temperature setting on both models and choose whichever produces more varied and creative descriptions