Fine-tuning open-source LLMs for domain-specific tasks

When prompt engineering is not enough

Prompt engineering has improved dramatically. With careful prompt design, you can get foundation models to perform remarkably well on many tasks. But there are cases where prompt engineering hits a ceiling.

Fine-tuning matters when:

• You need consistent output formatting that prompt engineering cannot guarantee
• The domain requires specialised vocabulary or reasoning patterns
• You need to reduce token usage (a fine-tuned model can accomplish in 200 tokens what a prompted model needs 2,000 tokens for)
• Latency matters and you need smaller, faster models

Choosing the right base model

For most fine-tuning tasks, we start with open-source models: Llama 3, Mistral, or Qwen 2.5. These models offer strong baseline performance with the flexibility to fine-tune on your own hardware.

The choice depends on your constraints: Llama 3 for general capability, Mistral for efficiency, Qwen 2.5 for multilingual tasks. All three can be fine-tuned on a single GPU with 24GB VRAM using LoRA adapters.

The fine-tuning process

Quality data matters more than quantity. We have found that 500-2,000 high-quality examples often outperform 50,000 noisy ones. The process:

1. Data preparation: Collect and format training examples in instruction-response pairs. Clean aggressively — remove duplicates, fix inconsistencies, and validate outputs.
2. Base model evaluation: Test the base model on your task to establish a baseline. This tells you how much improvement to expect from fine-tuning.
3. LoRA fine-tuning: Train LoRA adapters with rank 16-64, depending on task complexity. Full fine-tuning is rarely necessary and significantly more expensive.
4. Evaluation: Test the fine-tuned model against your baseline. Measure not just accuracy but also latency, token usage, and failure modes.

What we learned

The biggest lesson: fine-tuning is iterative. Your first training run will not be your last. Budget for 5-10 iterations, each informed by evaluation results from the previous one. The data quality improvements between iterations matter more than hyperparameter tuning.