Prompting vs RAG vs Fine-Tuning: A Decision Framework for LLMs

What each technique actually changes

Prompting changes the instructions and examples the model sees on each request. It covers role, tone, rules, output format, a few worked examples and the definitions of any tools the model may call. It is instant to change, needs no training data and is where every project should begin. Its limits are context length, cost per token and the fact that a model will follow long instruction lists less reliably than short ones.

RAG changes what the model knows at answer time by retrieving relevant passages from your documents or databases and placing them in the prompt. It is the right tool when the answer depends on private, changing or citable information. Fine-tuning changes the model's weights using examples of inputs and desired outputs. It is good at teaching consistent format, style, classification boundaries and domain phrasing, and poor at reliably adding facts; a fine-tuned model may produce a fact it saw in training, but it cannot cite it, and nothing updates it when the fact changes.

A concrete example makes the difference clear. Picture an assistant that drafts replies to insurance claim enquiries. Knowing the current excess for a given policy is a knowledge problem, so it belongs in retrieval from the policy system. Writing in the insurer's approved tone, with the mandatory disclosure in the right place, is a behaviour problem that good prompting usually solves. Classifying thousands of incoming emails a day into twenty claim types with a small, cheap model is where fine-tuning starts to pay.

• Prompting changes behaviour per request: instructions, examples, output schema and tools.
• RAG changes knowledge per request: fresh, private, citable context retrieved at answer time.
• Fine-tuning changes default behaviour: format, style, terminology and task-specific judgement baked into the weights.
• None of them fixes a vague task definition; that has to be solved before any of them will work.

Build the evaluation set before choosing anything

Every decision in this article depends on measurement, so the first deliverable on our LLM projects is an evaluation set, not a prototype. It is a few hundred real inputs with the expected output or a clear grading rule for each: the right answer and its source for a knowledge task, the correct label for a classifier, the required fields for an extraction. Without it, comparing a prompt change with a fine-tune is a matter of opinion.

Build it from production data where you can, and deliberately include the difficult cases: ambiguous questions, questions the documents do not answer, adversarial inputs and the long tail of rare categories. Agree pass criteria with the business owner in advance. Use automated grading where it is reliable, such as exact match, schema validation and citation checks, and calibrate any LLM-as-judge grading against human reviewers on a sample before trusting it.

• Start with 100 to 300 cases drawn from real traffic, then grow it with every production failure.
• Record why each case exists, so nobody deletes the hard ones to improve the score.
• Track quality, latency and cost per request together; a gain on one is often paid for on another.
• Run the full set on every prompt change, model upgrade and data refresh, as a release gate.

A decision path that works in practice

Begin with a strong prompt on a capable model and run the evaluation set. If the failures are about missing or outdated knowledge, such as wrong policy details, invented product specifications or answers that should cite a document, add retrieval. If the failures are about behaviour, such as inconsistent format, wrong tone, misread domain terms or a classifier that keeps blurring two categories, first try better instructions and more examples, then structured outputs, and only then consider fine-tuning.

Fine-tuning earns its place in a few recognisable situations: the prompt has grown to thousands of tokens of rules and examples and is still inconsistent; you need a small, fast model to match a large model on one narrow task; or a specialised output style must hold across very high volumes. Most production systems end up combining techniques: a fine-tuned or carefully prompted model that reads retrieved context and follows a fixed output schema. The order you add them in matters more than the final mix.

• Wrong or missing facts point to retrieval, not training.
• Inconsistent format points to structured outputs and examples before fine-tuning.
• A huge prompt doing a narrow, high-volume job points to fine-tuning a smaller model.
• Failures on every axis usually mean the task itself is badly specified.

What data each approach really needs

Prompting needs a clear task description and a handful of good examples, which a domain expert can write in an afternoon. RAG needs a corpus that is accessible, reasonably current and parseable: PDFs with tables, scanned forms, wiki pages and ticket histories all need different handling, and document-level permissions must travel with every chunk so the system never retrieves text a user is not allowed to see. Ingestion quality, not the vector database, decides most RAG outcomes.

Fine-tuning needs labelled examples of the exact behaviour you want, and quality matters far more than quantity. For format and style tasks, a few hundred to a few thousand carefully reviewed examples is a typical starting range; a larger set of inconsistent examples teaches the inconsistency. The examples must also be legally usable for training, which rules out some customer data and most third-party content. Keep a held-out portion that is never trained on, or the evaluation numbers will flatter the model.

Synthetic data can fill gaps, but only with review. Generating candidate examples with a stronger model and having domain experts correct them is a sound way to build a training set quickly; training on unreviewed synthetic output mostly teaches the smaller model the generator's habits and mistakes. The same discipline applies to RAG, where the most valuable data asset is often a set of real questions paired with the correct answer and the exact source passage.

Cost, latency and maintenance trade-offs

Prompting has the lowest upfront cost and the most visible running cost: every request pays for the instructions and examples again. Prompt caching on providers that support it reduces this substantially for long, stable prefixes, which makes well-structured prompts cheaper than they look. Long-context models can also take a small corpus whole, which is a legitimate alternative to RAG for a few hundred pages, but cost and latency grow with every token, and accuracy on details buried in the middle of a very long context still varies.

RAG adds an ingestion pipeline, an index, a retrieval step and usually a reranker. Retrieval typically adds tens to a few hundred milliseconds, and the real cost is operational: re-indexing when documents change, monitoring retrieval quality and handling permissions. Fine-tuning adds training runs, model versioning and, for open-weights models, serving infrastructure. It also creates a maintenance obligation: when the base model is upgraded or deprecated, the fine-tune has to be redone and re-evaluated.

In return, a fine-tuned small model can be much faster and cheaper per request than a large model with a long prompt. That trade only pays off at volume. Moving a narrow extraction task that runs millions of times a month from a large model with a long prompt to a fine-tuned small model with a short one can cut both latency and cost several times over; at a few thousand requests a month, the engineering effort is unlikely ever to be repaid. Estimate monthly request counts before committing, and budget for redoing the fine-tune when the base model changes.

Frontier APIs, hybrid gateways or private models

Technique and model hosting are separate decisions, and our Budget Calculator asks about the second one directly with three options. Frontier APIs give the strongest general reasoning and the fastest start; the work is in reviewing the provider's data retention, residency and contractual terms, and in planning for model version changes you do not control. For most first releases this is the right default.

Private models mean open-weights models served on your own infrastructure, typically with vLLM on GPUs in your cloud account or data centre. They suit regulated data, air-gapped environments and very high, steady volumes, and they are also where fine-tuning is most flexible. The cost is real engineering: GPU capacity planning, serving, monitoring and upgrades. Our MedOmni clinical RAG copilot took this route because patient records could not leave the hospital network.

A hybrid gateway sits between the two: one internal API that routes each request by sensitivity, complexity and cost, sending routine or sensitive work to a small or private model and escalating hard reasoning to a frontier model, with fallback when a provider is unavailable. It adds a component to run, and it keeps you from being locked into one vendor's roadmap.

Common mistakes we see

The most frequent mistake is fine-tuning to add knowledge: training a model on a policy manual and expecting accurate, current, citable answers. It produces fluent text that is confidently out of date. The second is building RAG for a problem that is really about behaviour, then tuning chunk sizes for weeks when the model simply needed clearer instructions and an output schema.

The third is over-engineering early: a multi-stage pipeline with a knowledge graph, rerankers and several agents before anyone has checked whether a clear prompt with the five most relevant documents would already pass. Start with the simplest version that could work, measure it, and let observed failures justify each new component. The remaining mistakes are about process rather than technique, and they are just as costly.

• Choosing a technique before building an evaluation set, then judging success by a few hand-picked demos.
• Fine-tuning on unreviewed historical outputs, which bakes existing errors and inconsistencies into the model.
• Ignoring document permissions in RAG, so the assistant can quote text the user should never see.
• Comparing options on quality alone without measuring latency and cost per request at expected volume.
• Treating the model as fixed; every provider retires versions, so re-evaluation has to be routine.

How we would approach this

We start every engagement with the evaluation set and a baseline prompt on a capable model, which usually takes the first week or two and answers the question of technique with evidence rather than preference. If knowledge is the gap, RAG Solutions & Knowledge Engines covers ingestion, hybrid retrieval and citations. If behaviour, cost or latency is the gap, LLM Integration & Fine-Tuning covers structured outputs, model routing and fine-tuning. Generative AI Solutions covers private serving of open-weights models when data cannot leave your environment.

For a first estimate, the Budget Calculator lets you compare the Frontier APIs, Hybrid gateway and Private models options against timeline and team size. If you already have sample data and a target use case, a scoping call is the quickest way to agree what to measure and what to build first. We will be direct if a well-written prompt on an existing model is all you need.