What RAG Actually Is
Retrieval-Augmented Generation is a pattern where, instead of relying solely on an LLM's pre-trained knowledge, you retrieve relevant documents from an external data source and include them in the prompt as context before the model generates a response. The retrieval step typically involves embedding the user's query, searching a vector database for semantically similar document chunks, retrieving the top-N most relevant chunks, and including them in the prompt alongside the user's question. The LLM then generates a response grounded in those retrieved documents. The key properties of RAG are that the knowledge is external and updatable: you can add new documents, remove outdated ones, and update existing content without retraining the model. The model itself stays the same. RAG also provides natural auditability: you can log which documents were retrieved for each response and verify that the answer was grounded in your source material. This is important for GDPR data subject requests and for trust in regulated industries like healthcare or legal services.
What Fine-Tuning Actually Is
Fine-tuning is the process of continuing the training of a pre-trained LLM on a new dataset, updating the model's weights to reflect patterns in your specific data. After fine-tuning, the model responds differently, it has internalised the style, tone, terminology, and patterns from your training data as part of its weights rather than having this information injected at inference time. Fine-tuning can adjust how the model writes (matching a specific brand voice or documentation style), what terminology it uses (medical, legal, or technical domain vocabulary), how it formats responses (always returning JSON in a specific schema), and what it knows at a baseline level (though this is less reliable for factual knowledge). Important distinction: fine-tuning is not the same as training from scratch. You are modifying an existing model, not building a new one. OpenAI, Anthropic (with limitations), and open-source model providers (Mistral, Llama) all offer fine-tuning pathways. Fine-tuning requires a labelled dataset, compute resources, and ongoing maintenance when the base model is updated.
Data Requirements and Preparation Cost
RAG requires your source documents in a format that can be chunked and embedded. PDFs, Word documents, web pages, database records, and structured data can all feed a RAG pipeline. The preparation cost is the chunking strategy, embedding generation, and vector database setup, typically a few days of engineering work with established libraries like LlamaIndex or LangChain. You need volume of source content, not labelled training examples. Fine-tuning requires high-quality labelled examples: pairs of input prompts and ideal output responses. Creating these examples is labour-intensive. A useful fine-tuning dataset for a customer service use case might require 500-5,000 examples, each manually crafted or carefully curated from existing interactions. Low-quality training data produces a fine-tuned model that has confidently learned the wrong patterns, which can be worse than the base model. The data preparation phase for fine-tuning is often the most expensive and time-consuming part, and it is a cost that recurs each time the base model changes.
Update Frequency and Knowledge Currency
RAG handles knowledge updates naturally and cheaply. Adding new documents to your vector store is a routine operation: chunk the document, generate embeddings, upsert to the vector database. Your LLM responses are immediately updated to reflect new information. This is essential for products where the knowledge base changes regularly: a legal research tool tracking new case law, a technical support bot keeping pace with product releases, or a financial analysis tool consuming regulatory updates. Fine-tuning produces a static model. When your knowledge base changes, the fine-tuned model does not automatically update. Re-fine-tuning requires a new training run, new compute costs, new evaluation, and a new deployment. For knowledge that changes frequently (more than quarterly), fine-tuning is operationally expensive to keep current. For knowledge that is stable (a legal analysis style that does not change, a specific output format, a brand voice), fine-tuning a specific behaviour that changes rarely makes more sense.
Cost at Scale and Production Considerations
RAG has predictable costs that scale linearly with usage: embedding generation per query, vector search (which is cheap), and LLM inference with a larger prompt (because you are including retrieved documents as context). The context window cost is the main variable: retrieving 10 document chunks at 200 tokens each adds 2,000 tokens to every prompt, increasing your LLM cost per request. For high-volume products, this context overhead is meaningful and should be optimised through good chunking strategy and retrieval precision. Fine-tuning has different cost dynamics. The training cost is a one-time or periodic expense (less with LoRA and other efficient fine-tuning methods). The inference cost can be lower than RAG because fine-tuned models can often produce good responses with shorter prompts (no document retrieval context needed). For extremely high-volume use cases where the specific behaviour the fine-tune achieves is consistent across all requests, fine-tuning can be more cost-efficient at scale. For most early-stage products, this is a premature optimisation.
Hallucination and Reliability
RAG reduces hallucination on knowledge questions by grounding responses in retrieved source material. When the model is told 'based on these documents, answer the following question', it is less likely to fabricate information because it has direct source material to reference. You can further constrain this by instructing the model to only use information from the provided context. This provides a level of verifiability: if the answer is wrong, you can trace which retrieved documents led to it. Fine-tuning does not reliably reduce hallucination on factual questions. A fine-tuned model internalises patterns from training data but can still confabulate facts with high confidence, particularly for questions outside the training distribution. Fine-tuning is better for style and format consistency than for factual accuracy. For products in regulated industries (healthcare, financial advice, legal) where factually incorrect responses carry legal or safety risk, RAG with source citations is almost always the more defensible architecture.
When Fine-Tuning Is the Right Choice
Fine-tuning earns its cost and complexity in specific, well-defined scenarios. If your product requires a highly consistent output format that is difficult to achieve reliably through prompting alone (a specific JSON schema, a particular document structure, a constrained vocabulary), fine-tuning on examples of that format can improve reliability. If your product needs to match a specific writing style that system prompts alone cannot capture (a brand voice with subtle stylistic patterns, a technical documentation style with unusual conventions), a fine-tuned model can produce more consistent stylistic outputs. If you are using a smaller, cheaper model (GPT-4o Mini, Claude Haiku) and want it to perform on a specific task at a quality level closer to a larger model, fine-tuning on high-quality task examples can close some of that capability gap. Do not fine-tune for factual knowledge: use RAG. Fine-tune for style, format, and task-specific behaviour.
Verdict
For most AI products, especially at MVP and early production stage, RAG is the right starting point. It is faster to implement, requires no labelled training data, keeps knowledge current without retraining, provides natural auditability for compliance purposes, and reduces hallucination on knowledge questions. Start with RAG, optimise your retrieval and chunking strategy, and evaluate whether fine-tuning is necessary for specific behaviours the RAG system cannot reliably achieve. Fine-tuning is a later-stage optimisation for teams with the data, compute budget, and evaluation infrastructure to do it properly. SpeedMVPs builds RAG pipelines as a standard component of AI MVPs for clients who need document-grounded responses. We implement fine-tuning when the client's specific output requirements cannot be met through prompt engineering and retrieval optimisation alone.