Question

Your team must fine-tune a large open-source language model for a specialized legal-summarization task. You have a single GPU with limited VRAM and a modest labeled dataset. Full fine-tuning of all model weights repeatedly runs out of memory, and you also need to keep the fine-tuning artifact small so that multiple task-specific adapters can be swapped in and out on the same base model in production. Which fine-tuning approach best meets these requirements?

Accepted Answer

Parameter-efficient fine-tuning with LoRA, training only low-rank adapter matrices while keeping the base model weights frozen

Answer

Full fine-tuning of all model parameters using gradient checkpointing to fit the model in memory

Answer

Few-shot prompting with retrieved examples instead of updating any model weights

Answer

Continued pre-training on a large unlabeled legal corpus before task training