The short answer

  • Prompt engineering when the model knows enough but is not doing what you want. Cheapest, fastest to change.
  • Retrieval (RAG) when the model lacks knowledge that is private, changes often or is too large to paste in.
  • Fine-tuning when you need a consistent behaviour, format or style at volume, and the knowledge involved is stable.

And before any of them: write down what "good" looks like and how you will test it. Both OpenAI and Anthropic put that step first in their own guides.

Prompt engineering first

OpenAI's model optimisation guide describes a loop: write evals (tests that score the model's output), prompt, measure, adjust, repeat. It says plainly that the prompt engineering process "may be all you need". Its basics are unglamorous: include the context the model needs, give clear instructions about the output, and show a few examples of correct answers.

Anthropic's prompt engineering overview starts in the same place, asking for success criteria and a way to test against them before any prompt work. It also notes that not every failing test is a prompting problem; sometimes latency and cost are easier to fix by choosing a different model.

Failure mode: the prompt grows into a fragile wall of rules that nobody dares touch. Evals are the cure, because they tell you whether an edit helped. Cost shape: nothing to build, but every instruction is re-sent and paid for on every call.

When the whole library fits

Before building retrieval, check whether you need it. In its 2024 write-up on contextual retrieval, Anthropic suggested that a knowledge base under 200,000 tokens, about 500 pages, can simply go into the prompt in full, and that prompt caching makes this fast and cheap.

Using today's prices, the sum is simple. Sending 200,000 tokens to Claude Sonnet 5.5 costs $0.40 a request at the $2 per million input rate. Read from the cache at $0.10 per million, the same tokens cost $0.02. For a small, stable library queried a few hundred times a day, that can beat running a retrieval pipeline. Long contexts carry their own quality risks, which we covered in context rot.

Retrieval-augmented generation

RAG fetches the most relevant passages from your documents at question time and adds them to the prompt. The name comes from a 2020 paper by Patrick Lewis and colleagues, who paired a language model with a searchable index of Wikipedia. Their stated motivations still apply: models are poor at precisely using the knowledge they hold, they cannot show where an answer came from, and their knowledge is hard to update. With retrieval you update the documents, not the model, and you can cite the passage used.

Failure mode: retrieval misses. If the right passage is not fetched, the model answers from memory or guesses, and does so fluently. Anthropic's contextual retrieval tests found that adding a short explanation of context to each chunk before indexing cut failed retrievals by 49%, and by 67% with a reranking step. Most RAG problems are search problems, not model problems.

Cost shape: an indexing pipeline, a vector database or search index, and extra tokens per answer for the retrieved text. Frameworks such as LangChain and LlamaIndex do much of the wiring; our LangChain vs LlamaIndex vs LangGraph comparison and the LLM frameworks guide help you choose. If your sources are web pages, a scraping API such as Context.dev turns them into clean text first.

Fine-tuning

Fine-tuning trains an existing model further on your examples. OpenAI's guide lists what it buys over prompting: more examples than fit in a single request, and responses that consistently follow a format or handle unusual inputs. It is about behaviour. It is a poor way to teach facts that change, because the knowledge is frozen at training time.

Full fine-tuning of a large model is expensive, which is why most teams use parameter-efficient methods. The 2021 LoRA paper by Edward Hu and colleagues showed that freezing the original weights and training small added matrices cut trainable parameters 10,000-fold and GPU memory three-fold compared with fully fine-tuning GPT-3 175B, with quality on par or better on the models tested. Libraries such as Hugging Face's PEFT package these methods; see our Hugging Face profile.

One change matters this month. OpenAI's documentation now says it is winding down its fine-tuning platform: it is closed to new users, existing users can keep creating training jobs "for the coming months", and fine-tuned models stay available until their base models are deprecated. For new projects, fine-tuning increasingly means open-weight models you host or rent, which adds operating work.

Failure mode: you train in yesterday's knowledge and lose the model when its base is retired. Cost shape: a curated dataset (the expensive part is people's time), training runs, and hosting if you use open weights.

A decision guide

  1. Can you test it? If not, stop and write ten to fifty real examples with the answers you expect.
  2. Does a better prompt or a different model pass the tests? If yes, you are done.
  3. Is the missing piece knowledge? If it fits in the context window, paste it in and cache it. If it does not, or it changes daily, build retrieval, and spend your effort on search quality.
  4. Is the missing piece behaviour at volume? A fixed output format, a house style, a classification run millions of times. Then fine-tuning may pay, ideally on a small open model.
  5. Is it both? Combine them: a tuned model for the format, retrieval for the facts.

And sometimes the answer is none of them. If the task needs guaranteed correctness, such as tax sums or legal deadlines, ordinary code or a lookup table beats any model. For citable research answers, a search tool like Perplexity may be all you need. Browse the AI infrastructure and research and analysis directories for the building blocks, and our notes on the Claude Code architecture paper for how one serious agent manages its context.

Sources