RAG, fine-tuning, or a small model? A plain guide to a confusing choice

Ask three vendors how to build your AI feature and you will often get three answers: retrieval, fine-tuning, or a small custom model. They are not interchangeable, and choosing the wrong one can cost you months. Here is the plain version.

RAG: give the model your knowledge

Retrieval-augmented generation leaves the model as it is and feeds it the right information at the moment it answers. Your documents stay in a searchable store, and the relevant pieces are pulled in with each question. It fits when your knowledge changes often, when answers have to cite a source, and when you need to add or remove information without retraining anything. Most business question-answering should start here.

Fine-tuning: teach the model a behaviour

Fine-tuning adjusts the model itself on examples of the work you want. It teaches form more than facts: a consistent tone, a strict output format, a narrow task done the same way every time. Reach for it when you have good examples and a behaviour that prompting keeps getting almost, but not quite, right. Reach for it to store facts and you will pay handsomely to bake in knowledge that goes stale.

Small models: own the whole thing

Sometimes the right answer is a smaller model that runs cheaply, quickly, and entirely under your control. When a task is narrow, high-volume, privacy-sensitive, or has to work on a poor connection, a small or distilled model can beat a frontier one on the measures that actually matter for that job: cost, speed, and control.

• ▸Knowledge that changes often and needs citations: start with RAG.
• ▸A consistent tone, format, or specialised task: fine-tune.
• ▸Narrow, high-volume, private, or offline: a small model.
• ▸A genuinely hard reasoning problem: keep the frontier model, and route only the hard parts to it.

The honest answer is usually a combination

Real systems rarely pick just one. A common shape is a small model handling the routine traffic, RAG supplying current knowledge with sources, and a frontier model held in reserve for the few questions that truly need it. The skill is not loyalty to a single technique. It is matching each part of the problem to the cheapest tool that does it well.

The wrong architecture is not a small mistake. It is months of work pointed in the wrong direction.

If someone recommends an approach before they understand your data, your accuracy bar, and your constraints, slow down. The choice should come from the problem, not from habit.