Question

You fine-tuned a base language model on a large domain-specific dataset for a legal-document assistant. After deployment, evaluation shows the model now performs well on legal tasks but has significantly degraded on general reasoning and instruction-following tasks it previously handled. You must re-run the customization to preserve general capabilities while still gaining domain accuracy, using minimal compute. Which approach best addresses this problem?

Accepted Answer

Use a parameter-efficient fine-tuning method such as LoRA with a lower learning rate and fewer epochs, keeping the base model weights frozen.

Answer

Increase the number of training epochs on the full-parameter fine-tune to force the model to relearn general tasks.

Answer

Raise the learning rate on the full-parameter fine-tune so the model converges faster on the legal dataset.

Answer

Discard the domain dataset and rely solely on retrieval-augmented generation with the untuned base model.