Prompt, retrieve or fine-tune? Getting a model to use your material

Short answer

First decide what is wrong. If the model does not know your facts, give it your material: in the prompt if it fits, through retrieval if it does not fit or changes often. If it knows the facts but gets the format, tone or structure wrong, show it examples. Fine-tune only for behaviour, only after the cheaper rungs fail on a test set, and only with a few hundred good examples. In October 2026 the three largest vendors offer little fine-tuning anyway.

Who this is for. Anyone setting up an assistant, a project or an API call that must use their own documents, style or format. The first three rungs need no code.

Name the problem first

OpenAI's accuracy guide splits the problem in two. You optimise the context when "the model lacks contextual knowledge because it wasn't in its training set". You optimise the model when it "is producing inconsistent results with incorrect formatting" (OpenAI, vendor).

Many cases are both. Fix knowledge first, because a format fix cannot rescue a wrong fact.

The ladder

Try the rule on seven failures from an illustrative help-desk assistant. Sort each one before you look at the rung it points to.

Rung 0: the test set

Write 20 questions your users really ask. Include a few that need two documents, a few whose answer is buried deep in a long file, and two or three whose answer is "not in our material". For each, record the right answer and where it lives. How to write pass rules and judge them is in Test any new model on your own tasks. OpenAI's guide starts the same way: "a good prompt with an evaluation set of questions and ground truth answers".

id,question,expected,source_doc,source_passage,rung,retrieved_ok,answer_ok,note
<Q_ID>,"<QUESTION>","<EXPECTED>",<SOURCE_DOC>,"<SOURCE_PASSAGE>",<RUNG>,<RETRIEVED_OK>,<ANSWER_OK>,"<NOTE>"

Replace the following:

• <Q_ID>: a short code, such as Q07.
• <QUESTION>: the question as a user would type it.
• <EXPECTED>: the answer, or the pass rule for it.
• <SOURCE_DOC> and <SOURCE_PASSAGE>: the file and the sentence that holds the answer, or none.
• <RUNG>: 1 to 4.
• <RETRIEVED_OK>: Y or N at rung 3, n/a before it.
• <ANSWER_OK>: Y or N.
• <NOTE>: why it failed, in a few words.

Worked example (illustrative). A small firm wants an assistant that answers staff questions from its 40-page handbook. Four of its 20 rows, after runs at rungs 1 and 2:

id,question,expected,source_doc,source_passage,rung,retrieved_ok,answer_ok,note
Q01,"How many days of leave do I get?","25 days plus public holidays",handbook.pdf,"Full-time staff receive 25 days...",1,n/a,N,"said 20, general knowledge"
Q01,"How many days of leave do I get?","25 days plus public holidays",handbook.pdf,"Full-time staff receive 25 days...",2,n/a,Y,""
Q09,"Can I carry leave over and claim travel for the same trip?","Up to 5 days carry-over; travel needs prior approval",handbook.pdf; expenses.pdf,"Up to 5 days may be carried... / Travel must be approved...",2,n/a,N,"answered leave only, two-document question"
Q17,"Do we offer a cycle-to-work scheme?","Not in our material",none,none,2,n/a,N,"invented a scheme"

What it showed: Q01 was a knowledge failure that rung 2 fixed. Q17 is the confident invented answer the rung 2 check warns about, so they said in the prompt that "not in the documents" is acceptable. Q09 needs a prompt that asks for quotes from each document before the answer. With 2 of 20 failing at rung 2, the rule in the ladder says stay on rung 2 and fix the prompt, not climb.

Rung 1: a better prompt

OpenAI says "Prompt engineering is typically the best place to start". State the task, the audience and the format, and add two or three examples of a good answer. The basics are in Prompting in one page and Tell it who you are.

Rung 2: put the material in the context

Current Claude models, OpenAI's flagship models and many Gemini models take about a million tokens (see the dated box). Anthropic suggests that a knowledge base under 200,000 tokens, about 500 pages, can go into the prompt whole, with no retrieval, and that prompt caching makes this faster and cheaper (Anthropic, vendor).

Large is not the same as reliable. Anthropic's docs say "more context isn't automatically better", because accuracy and recall degrade as inputs grow. In a peer-reviewed study, models did best when the relevant passage sat at the start or end of the input, and worse when it sat in the middle (Liu et al., 2024, benchmark). Google notes that when you need several specific facts at once, "the model does not perform with the same accuracy" (Google, vendor).

What the vendors advise:

• Put long documents at the top and the question at the end. In Anthropic's tests this improved answers by up to 30% (Anthropic, vendor). Google gives the same advice.
• Wrap each document in tags with its source, and ask for the relevant quotes before the answer.
• For repeated calls over the same material, use caching. Google calls it "the primary optimization" for long context.

Rung 3: retrieval

Retrieval fetches the passages that look relevant and answers from them. Move here when the material does not fit, changes too often to re-paste, or costs too much to send whole. Claude Projects on paid plans switch to retrieval by themselves near the context limit. OpenAI and Google offer file search tools in their APIs.

Retrieval adds a second way to fail. In OpenAI's words: "You can supply the wrong context, so the model can't possibly answer". So score two things for every question:

1. Was the right passage retrieved? Compare what came back with source_passage. OpenAI's file search returns the results only if you ask, with include: ["file_search_call.results"] (OpenAI, vendor). Gemini's returns citations to the passages it used.
2. Was the answer right, given what was retrieved?

Retrieval itself can be improved. Anthropic added a short explanatory context to each chunk before indexing. On its test sets that cut the top-20 retrieval failure rate by 35%, by 49% when combined with keyword search, and by 67% with reranking added (Anthropic, vendor data).

Rung 4: fine-tuning, for behaviour only

Fine-tuning trains a model on examples of the output you want. OpenAI's guide places it on the behaviour side, for consistency and efficiency, not for missing knowledge. Use it when rungs 1 to 3 pass on facts but the format or style still drifts over many calls, or when you want a smaller model to do what a larger one does with a long prompt.

The data rules, from OpenAI's fine-tuning guide (OpenAI, vendor):

• The minimum is 10 examples. OpenAI reports improvements from 50 to 100, and recommends starting with 50 well-crafted ones.
• "If 50 examples have no impact, rethink your task or prompt before adding training data."
• Hold data back before training: "This holdout data acts as a control group when you use it for evals."

Dated box, read 2026-10-10. Offers change often.

How to tell it worked

• Every one of the 20 rows has an expected answer and a source passage, or none.
• Each failure carries a note that names it as knowledge or behaviour.
• You stopped at the lowest rung that fails fewer than 3 of the 20, and you can say why you did not climb.

Limits

Try this next

1. Write the 20 questions and their source passages. (45 minutes, our estimate)
2. Run rungs 1 and 2, and sort the failures into knowledge and behaviour. (30 minutes, our estimate)
3. Keep the file and re-run it when you change model: Test any new model on your own tasks.

Sources

1. OpenAI. Optimizing LLM accuracy. Vendor documentation. Link
2. OpenAI. Supervised fine-tuning; Model optimization; Deprecations. Vendor documentation. Fine-tuning · Model optimization · Deprecations
3. OpenAI. File search; Models; Projects in ChatGPT. Vendor documentation. File search · Models · Projects
4. Anthropic. Introducing Contextual Retrieval, 2024. Vendor data. Link
5. Anthropic. Long context prompting tips; Context windows; Models overview; RAG for Projects. Vendor documentation. Tips · Context windows · Models · Projects
6. Google. Long context; File Search; Gemini 3.8 Flash and 3.1 Pro Preview model pages; Model tuning. Vendor documentation. Long context · File Search · 3.8 Flash · 3.1 Pro Preview · Tuning
7. Amazon Web Services. Supported models for model customization. Vendor documentation. Link
8. Liu et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 12, 2024. Peer-reviewed benchmark. Link

Changes

1. v1.010 Oct 2026first version