Choosing prompting, RAG, and fine-tuning

Section 01

Three ways to change model behavior

When an LLM system does not perform well enough, you have three levers: change the prompt, add retrieval, or change the model weights. Each lever changes different things, at different costs, with different reversibility. Choosing the wrong lever wastes time and money.

This lesson compares the three approaches so you can make that choice based on what your system actually needs to change — not on what is fashionable.

Section 02

Three tools on the workbench

The instruction jig (prompting) shapes how the model approaches the task without changing what it knows. It is cheap, fast, and immediately reversible. But it cannot add knowledge the model does not have.

The archive lens (RAG) gives the model access to external evidence at inference time. It adds fresh, specific knowledge without retraining. But it requires infrastructure (indexing, retrieval, verification) and adds latency.

The shaping die (fine-tuning) permanently alters the model's weights to change its behavior. It can teach new patterns, adjust style, and improve specific task performance. But it is expensive, slow, requires curated training data, and is hard to reverse.

Where the analogy breaks: these tools are not mutually exclusive. A production system typically uses all three — prompting for task framing, RAG for external knowledge, and fine-tuning for persistent behavior changes that prompting alone cannot achieve.

Section 03

What each lever changes

Prompting changes the model's instructions at inference time. It controls format, tone, scope, and approach. It is the first lever to try because it costs nothing but time. If the model knows the answer but produces it in the wrong format, prompting fixes it. If the model does not know the answer, prompting cannot help.

RAG changes what the model can reference at inference time. It adds external evidence to the context window. It helps when the model needs specific, current, or proprietary information not in its training data. It does not change the model's fundamental capabilities.

Fine-tuning changes the model's weights. It helps when the model needs to learn a persistent pattern: a specific output format, a domain-specific vocabulary, a style, or a behavioral tendency. It is the most expensive lever and should be tried last, after prompting and retrieval have been exhausted.

Section 04

Comparing by requirement

Different problems call for different levers:

• Freshness (needs current data): RAG. Fine-tuning cannot keep up with changing data.
• Format consistency: prompting first, then fine-tuning if prompting is unreliable.
• Domain vocabulary: fine-tuning for persistent terminology; RAG for document-specific terms.
• Latency budget: prompting is free; RAG adds retrieval latency; fine-tuning is inference-equivalent.
• Privacy: RAG keeps data outside the model; fine-tuning bakes data into weights (harder to audit or remove).
• Reversibility: prompting is trivially reversible; RAG requires re-indexing; fine-tuning requires retraining.

Section 05

Combining approaches

Combine approaches only when each addresses a separate, measured requirement. A common stack: fine-tuned model for domain vocabulary and output style, RAG for current evidence, prompting for task-specific instructions. Each layer must earn its place through evaluation — do not add complexity without evidence that it helps.

Section 06

Common mischoices

Fine-tuning for knowledge: the most expensive and least effective way to give a model current information. Use RAG.

RAG for style: retrieval does not change how the model writes. If you need a consistent output style, use prompting or fine-tuning.

Prompting for capability: if the model fundamentally cannot do the task (e.g., it lacks the reasoning depth), no prompt will fix it. You need a different model or a different architecture.

Stacking all three without evaluation: adding complexity without measuring improvement. Each layer should show a measurable quality gain on your evaluation set.

Section 07

Unscored lab: diagnose the right lever

For each of the following problems, identify the correct lever (prompting, RAG, fine-tuning, or a combination) and justify your choice.

1. The model knows the answer but returns it in paragraph form instead of JSON.
2. The model needs to reference the company's current HR policy document.
3. The model should consistently write in the company's brand voice across all responses.
4. The model needs to understand a specialized vocabulary of 200 domain terms.
5. The model sometimes answers questions it should refuse (out-of-scope requests).
6. For each: what would you try first? What evidence would you need to justify a more expensive approach?