RAG or fine-tuning: which should you choose for your business?

In 90% of business cases, RAG is the right first choice. Fine-tuning answers a different problem: it teaches the model a behaviour (a style, an output format), not your knowledge.

The question comes up on every AI assistant project: "should we train a model on our data?" Behind it there is often a confusion fuelled by marketing: people imagine that a model "trained on our documents" will know them by heart. In practice, that is not how it works, and choosing the wrong approach is expensive. Here is how I decide, criterion by criterion, with the real costs of both options.

In short

• To answer from your documents (customer questions, support, internal knowledge base), RAG is the right tool in the vast majority of cases: in my opinion, 9 business projects out of 10.
• On budget: an operational RAG is built for a few thousand euros; a serious fine-tuning first requires an annotated dataset, often several hundred to several thousand examples, before even the first round of training.
• Updating: with a RAG, adding a document takes a few seconds; with fine-tuning, any change in knowledge requires a complete retraining.
• Traceability: a RAG cites the source of every answer; a fine-tuned model cannot say where what it asserts comes from.
• The real territory of fine-tuning: behaviour (tone, format, high-volume classification), not knowledge.

Two techniques, two different problems

RAG (Retrieval-Augmented Generation) consists of searching for the right passages in your documents at the moment of the question, then providing them to the model so it writes its answer from them. The model itself does not change: what changes is what you give it to read. Your data stays in your database, consulted on demand, and each answer can cite the original document. It has been the reference technique for documentary assistants since the foundational paper by Lewis et al. in 2020. I explained the mechanism in detail in what is RAG.

Fine-tuning consists of retraining an existing model on examples of question-answer pairs, to durably modify its behaviour. You show it hundreds or thousands of examples of the expected result, and it adjusts its internal weights to reproduce that behaviour. It is powerful for learning a tone, an output format or a repetitive task. But it is a poor vehicle for knowledge: the learned facts mix in with the rest of the model, with no traceability, and become obsolete as soon as your data changes.

The classic confusion: believing that fine-tuning a model on your documents will make it capable of restituting them faithfully. In reality, a fine-tuned model can paraphrase, mix, forget, and above all it cannot say "I don't know" about what it has not seen. For factual reliability, it is RAG, with its safeguards, that does the work (see how to stop AI from making things up).

The decision table

Here are the criteria I use to decide, in the order in which they eliminate an option fastest:

• Data freshness. RAG: real time, a document added to the base is usable immediately. Fine-tuning: knowledge is frozen at the date of the retraining; any update requires a new training session. If your content moves (prices, procedures, catalogue), this criterion alone decides.
• Traceability and citation. RAG: yes, each answer points to the original document and passage, so an error is verified in one click. Fine-tuning: no, the model restitutes what it has absorbed, without being able to say where it comes from. In business, that is often a dealbreaker.
• Initial cost. RAG: a few thousand euros for an operational assistant on a defined corpus. Fine-tuning: you first have to build and annotate a clean dataset (the most costly item, often underestimated), then pay for the training (GPU or API by training token), then evaluate, then start again because the first attempt is rarely the right one.
• Update cost. RAG: marginal, you add or replace documents. Fine-tuning: each significant change in knowledge or expected behaviour replays the whole cycle of data, training, evaluation.
• Confidentiality. RAG: your documents stay in your database, only the relevant passages travel to the model at the moment of the question; everything can be hosted in Europe, or even self-hosted. Fine-tuning: your data goes into the training set; depending on the provider, the resulting model is hosted on their side. Both can be compliant, but RAG offers control that is simpler to audit.
• Tone and style. This is the only criterion where fine-tuning clearly wins: to impose a consistent writing style or a strict output format, showing examples to the model works better than piling instructions into the prompt.
• Volume of data required. RAG: your documents as they are suffice, even a few dozen pages. Fine-tuning: you need quality annotated examples, in quantity; a small, poorly annotated corpus produces a model worse than the base model.

The cases where fine-tuning wins

Let's be honest: there are situations where fine-tuning is the right choice, and it would be dishonest to sell RAG everywhere.

• A consistent writing style. You generate thousands of texts that must follow a precise charter (product sheets, standard letters): a few hundred well-chosen examples teach the model the exact tone, better than a long prompt.
• Very high volume classification. Sorting tens of thousands of tickets, e-mails or documents per day: a small model fine-tuned for this single task is faster, more consistent and far cheaper at inference than a large generalist model called every time.
• Latency and cost in massive inference. Beyond a certain call volume, replacing a large model with a small fine-tuned specialised model reduces the bill and the response time. It is an optimisation calculation, relevant when the product is already running.
• An ultra-specialised domain with an annotated corpus. Medical, legal or industrial vocabulary very far from everyday language, and an annotated dataset that already exists: fine-tuning helps the model speak the language of the trade.

What these cases have in common: you teach the model how to do something, not what to know. As soon as the question is "answer correctly from our documents", you come back to RAG.

And often: both

It is not a duel. The architectures that age well combine the two: RAG for the facts, a model (sometimes fine-tuned) for the form. RAG goes and fetches the relevant passages and guarantees freshness and traceability; a small tuned model puts the answer in the right format, in the right tone. You get answers that are correct, sourced, and formatted exactly as the business expects. But you only get there after having made RAG work on its own: starting with fine-tuning means optimising the form before having the substance.

How much it costs

On the market side, a business RAG project generally costs from a few thousand euros for an assistant on a defined corpus, to several tens of thousands for complex deployments (multi-source, high volume, sovereignty constraints). A serious fine-tuning project rarely starts under tens of thousands of euros all in, because the dominant item is not the GPU: it is building the annotated dataset and the evaluation iterations. Fine-tuning APIs (OpenAI, Mistral) charge for the training by token, which seems modest, but that price covers neither the data preparation nor the successive retrainings.

At Buildoto, I do not sell fine-tuning: I deliver a RAG chatbot at a fixed price, 2 640 € excl. tax, based on the system that runs in production on BeForBuild.com, my B2B SaaS. The RAG is adapted to your documents (chunking, hybrid search, citations, "no source, no answer" clause), and the price is known before starting. If your need really is fine-tuning, I tell you so in the audit and point you elsewhere, it is not my line.

Where to start

1. A free 30-minute audit. We look at your real need: is it about knowledge (your documents) or behaviour (tone, format, classification)? This single question eliminates half the wrong leads.
2. A measurable scope. One corpus, one use case, real test questions. We measure the rate of correctly sourced answers before extending.
3. RAG first, optimisation later. If, once RAG is in production, a need for style or volume justifies a targeted fine-tuning, we add it knowingly, on real usage data.

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, 2020)
• OpenAI · official documentation for supervised fine-tuning
• Mistral AI · fine-tuning API documentation