Questions & answersRAG or fine-tuning: what fits company knowledge?
Two ways to teach a language model company knowledge. One is almost always the right start.
For company knowledge, RAG is almost always the right start: the model answers from your documents, with source references, and new documents are available immediately. Fine-tuning changes the model itself and pays off for style, format and terminology, not for facts. Both can be combined, but fine-tuning without RAG answers no question about documents created after training.
When does RAG make sense?
Whenever the model should answer questions about content that changes or that it cannot know: manuals, contracts, work instructions, ticket histories. RAG retrieves the matching passages for every question and displays them. New documents are available immediately, withdrawn ones disappear. That is the normal case for company knowledge.
When does fine-tuning make sense?
When it is not about facts but about behaviour: the model should answer in a fixed format, hit the house style, use terminology correctly or sort inputs into fixed categories. That requires a few hundred to a thousand good examples and a training run. Fine-tuning is also worthwhile when a small model should be specialised for a narrow task to save hardware.
When does neither make sense?
When a good model already does the job with a clear instruction. Many cases fail not because of the model but because of unclear requirements. First check whether a prompt is enough, then RAG, then fine-tuning.
Prerequisites
- RAG: documents in readable form, a decision on which versions apply, a permissions concept and a vector database such as ChromaDB.
- Fine-tuning: a clean dataset of examples, compute for training, an evaluation method and the willingness to repeat training when things change.
Benefits and limits compared
- Currency. RAG immediately, fine-tuning only after new training.
- Traceability. RAG names sources, fine-tuning does not.
- Permissions. RAG can filter per knowledge base. A fine-tuned model knows everything it learned, for every user.
- Behaviour. Fine-tuning shapes style and format more reliably than any instruction.
- Effort. RAG is productive in weeks, fine-tuning needs dataset, training and evaluation.
Example
A vendor wanted its model to know product data and thought of fine-tuning. The product data changed monthly. With RAG on the data sheets the system was productive in three weeks and always current. Only later did a small fine-tuning come in so answers kept the customer service's fixed format. Both together solved what neither would have solved alone.
Frequently asked
What is the difference between RAG and fine-tuning?
RAG retrieves matching passages from your documents for every question and lets the model answer from them. Fine-tuning retrains the model with examples so it adopts style, format or terminology. RAG changes the data the model sees. Fine-tuning changes the model.
Can fine-tuning learn facts?
Only unreliably. A fine-tuned model remembers approximately, not exactly, and knows no sources. For facts, figures and valid versions, RAG is the right method.
When is fine-tuning still worth it?
When the model should master a certain tone, a fixed output format or specialist vocabulary, for example for classification with fixed categories or for answers in the house style. It requires a few hundred to a thousand good examples.
Do both work together?
Yes. A lightly fine-tuned model for format and tone that accesses current documents via RAG. In practice you start with RAG and add fine-tuning only if a concrete problem remains.
Conclusion
RAG for knowledge, fine-tuning for behaviour. Anyone who wants to answer questions about their own documents starts with RAG. Fine-tuning is added when style, format or terminology remain a concrete problem. In that order, not the other way round.
Related questions: What does RAG mean in an enterprise context?, Can AI search company PDFs?, Which open-source AI is suitable for companies?
Enterprise knowledge with AI
Private knowledge systems that answer questions about your documents with source references. Locally or in an EU environment.
To the solution: enterprise knowledge