Fine-tuning an LLM and retrieval-augmented generation are routinely presented as competing options. They address different problems, and most teams reach for one when they needed the other — or when they needed neither.
The distinction in one line
RAG changes what the model knows at the moment of answering, by retrieving relevant information and including it in the prompt.
Fine-tuning changes how the model behaves, by adjusting its weights on examples of the behaviour you want.
If the problem is “the model does not know about our products”, that is knowledge, and RAG addresses it. If the problem is “the model answers in the wrong format or tone”, that is behaviour, and fine-tuning addresses it — though prompting often does too.
Try prompting first
A meaningful share of problems that teams plan to solve with fine-tuning are solved by a better prompt. Clear instructions, a few well-chosen examples, an explicit output schema and a structured format will get most tasks to acceptable quality.
This matters because prompting costs nothing to change. A fine-tuned model is a dependency: it must be retrained when the base model updates, versioned, evaluated and hosted. Committing to that before exhausting prompting is a common and expensive mistake.
The reasonable sequence is prompt, then prompt with retrieval, then fine-tune only if a measured gap remains.
When RAG is the answer
RAG fits when the model needs information it was not trained on, particularly when that information changes.
Internal documentation, product catalogues, support knowledge bases, customer records, recent events, anything with access controls — all of these suit retrieval. The content lives outside the model, so updating it is a matter of re-indexing a document rather than retraining anything.
Two further advantages matter in production. Citations: because you know which documents were retrieved, you can show the user where an answer came from, which is often a requirement. And access control: retrieval can be filtered by what the user is permitted to see, which is impossible once information is baked into weights.
The infrastructure needed is covered in our piece on vector databases.
When fine-tuning is the answer
Fine-tuning fits a narrower set of problems, and it fits them well.
Consistent output format that prompting cannot reliably enforce, particularly for structured generation at high volume.
A specific style or voice that would take a very long prompt to describe and that must be consistent across thousands of outputs.
A specialised classification or extraction task where you have thousands of labelled examples. A small fine-tuned model frequently beats a much larger general one here, at a fraction of the cost per call.
Latency and cost reduction. Fine-tuning a small model to match a large model’s behaviour on one narrow task can cut cost per request substantially — often the strongest commercial argument for it.
They are not mutually exclusive
A common production pattern uses both: fine-tune a model to follow your output format and domain conventions reliably, then use retrieval to supply the facts it needs for each specific request.
The fine-tuning handles behaviour that is stable; the retrieval handles knowledge that changes. Trying to make either do the other’s job is where most of the disappointment comes from.
What fine-tuning actually requires
Teams underestimate the data work.
- Examples. Hundreds at minimum for behavioural adjustment, thousands for a specialised task. Quality matters more than quantity, and inconsistent examples teach inconsistency.
- A held-out evaluation set that the model never trains on, with a defined metric.
- A baseline. Measure the prompted model’s performance first, or you cannot tell whether fine-tuning helped.
- A retraining plan. Base models are deprecated. A fine-tuned model is a thing you maintain, not a thing you build once.
Evaluation, which is the hard part either way
Neither approach can be tuned without a way to measure quality, and this is where most projects are weakest.
Build a test set of representative inputs with expected outputs or clear grading criteria, covering the easy cases, the hard cases and the ones where the right answer is “I do not know”. Run it on every change. Where outputs are open-ended, a combination of automated grading and periodic human review works better than either alone.
For RAG specifically, measure retrieval and generation separately. If the right document was never retrieved, no amount of prompt work will fix the answer, and conflating the two stages hides where the problem is.
Cost comparison
RAG costs more per request — retrieval plus a longer prompt — and almost nothing to update. Fine-tuning costs a training run plus ongoing hosting, and produces cheaper, shorter requests.
At low volume, RAG is cheaper overall. At high volume on a narrow task, a fine-tuned small model usually wins on total cost. The crossover depends heavily on request volume and on how often the underlying information changes, so it is worth modelling rather than assuming.
The decision in order
A sequence that avoids most of the wasted effort.
- Define the failure precisely. Not “the output is not good enough” but “it invents product names” or “it ignores the required JSON schema in 8% of cases”. The category of failure determines the remedy.
- Build the evaluation set before trying anything. Without it every subsequent step is unmeasurable.
- Improve the prompt. Explicit instructions, examples, an output schema, and a clear statement of what to do when the answer is unknown.
- Add retrieval if the failures are factual or concern information the model could not have.
- Consider a smaller or larger model. Sometimes the fix is simply a more capable model, and that takes an afternoon rather than a project.
- Fine-tune if a measured behavioural gap remains, or if cost at volume justifies replacing a large model with a specialised small one.
Teams that start at step six without doing steps one and two frequently produce a fine-tuned model that is no better and considerably harder to maintain.
Keeping it working over time
Both approaches degrade quietly, and the failure is rarely dramatic.
For retrieval, the corpus drifts: documents are added, edited and deleted, and an index that is not kept in sync starts returning stale answers confidently. Schedule re-indexing, and monitor the age of the oldest document in the index.
For fine-tuned models, the base model is eventually deprecated and the distribution of real inputs shifts away from the training examples. Re-evaluate on a schedule rather than waiting for complaints, and keep the training data in version control so retraining is routine.
In both cases, log a sample of real inputs and outputs in production and review them periodically. The gap between the evaluation set and what users actually send is where most quality problems live.
Frequently asked questions
Can fine-tuning teach a model new facts?
Poorly and unreliably. Facts learned through fine-tuning cannot be cited, cannot be access-controlled and cannot be updated without retraining. Use retrieval for facts.
How much data is needed to fine-tune?
For behavioural adjustment, often a few hundred high-quality examples. For specialised classification or extraction, several thousand. Consistency across examples matters more than volume.
Does a larger context window remove the need for RAG?
No. Large contexts are expensive, slower, and attention quality degrades across very long inputs. Retrieval sends only the relevant portion, which is cheaper and usually more accurate.
What is the most common RAG failure?
Retrieval, not generation. Poor chunking, a mismatched embedding model or missing metadata filters mean the right content is never provided, and the model then answers from general knowledge or declines.
