AI Tools Field GuideRAG done properly
Retrieval finds the passages that answer a question and puts those in the prompt, and it beats attaching a document once the corpus outgrows the context window or every answer needs a citation.
Why it exists
A model's knowledge is frozen at its training cutoff and general to the world. Your documents are current and specific to your command. The context window is finite, so you cannot hand the model everything you own on every question. Retrieval-augmented generation (RAG) closes that gap: at question time a search finds the passages that matter, those passages go into the prompt with "answer from these", and the model writes from them with citations you can open. The model is not retrained and nothing about it changes. What changes is what it reads.
The pipeline, step by step
| Step | What happens | Where it goes wrong |
|---|---|---|
| Ingest | Convert PDF, DOCX and slides to text; keep the title, date, section, and markings as metadata | Scanned pages, tables and multi-column layouts come out scrambled. Look at the text you got, not the file you gave. |
| Chunk | Split the text into pieces of a few hundred tokens, overlapping a little, cut on structure (headings, paragraphs) rather than on a count | A chunk boundary through the middle of a table or a numbered requirement leaves the model half a fact |
| Embed | A model turns each chunk into a vector, a list of numbers placing it by meaning; the same model later embeds the question | Acronyms and unit-specific terms embed poorly; "1650" and "award package" may land far apart |
| Index | Store the vectors; in practice also a keyword index, and search both | A vector-only index misses exact identifiers: instruction numbers, names, dates |
| Retrieve | Find the nearest chunks to the question, take the top few, and often re-rank them with a second model | Too few chunks and the answer is missing; too many and the model reads noise |
| Generate | The prompt says: answer from these chunks, cite them, and say when they do not contain the answer | The model answers from training when the chunks fall short, unless told not to |
Two settings do most of the work. Chunk size: a few hundred tokens, cut on headings, with the heading kept in every chunk so a passage still says which instruction and which section it came from. Hybrid search: vectors for meaning, keywords for exact strings, because a question about "OPNAVINST 5350.4" must match those characters and no embedding guarantees it.
When it beats a long context, and when it does not
Modern windows are large. For one document used once, attaching it is the right answer: the model sees the whole thing, and there is no pipeline to build or break. Retrieval wins when one of these is true:
| Condition | Why retrieval wins |
|---|---|
| The corpus is bigger than the window, or would cost too much to send every time | Only the relevant few thousand tokens go in |
| The documents change | Re-index the changed file; nothing else moves |
| Many people ask many questions | The index is built once and shared |
| Every answer must cite its source | Citations fall out of the pipeline for free |
| The material in the middle of a long context is used less reliably | Retrieval puts the right passage near the question (evidence) |
Long context wins when the question needs the whole document at once: cross-referencing sections, checking a document against itself, summarizing the lot. Retrieval hands the model fragments; some questions need the shape.
Uploaded files and document retrieval are part of most chat platforms now; OpenAI, for one, documents a file search tool that does exactly the pipeline above under the hood (OpenAI docs). Whether a given tool loads your whole attachment or retrieves passages from it is the one question to ask, because it decides whether the model saw all of your document or the parts a search chose. Beyond that, a retrieval system over your own corpus is an API project for your data shop.
The four failures
- The wrong chunk. The search returned something plausible and beside the point, and the model answered from it with full confidence. Cause: vocabulary mismatch, vector-only search, a question phrased unlike the documents. Fix: hybrid search, a re-ranker, and a test set (below).
- Half a chunk. The right passage was cut at the wrong place; the requirement's second clause lives in the next chunk, which was not retrieved. Fix: cut on structure, overlap, keep headings.
- The model went beyond the chunks. The passages did not contain the answer, so it supplied one from training. Fix: the instruction "if the passages do not answer this, say so", and a check that every claim has a citation that exists.
- The stale index. The instruction was superseded in May and the index was built in April. Fix: re-index on change, show the document date in every answer.
There is a fifth that belongs to security rather than accuracy: a document in the corpus contains instructions, and retrieval delivers them to the model as if you had typed them. See prompt injection. Retrieval widens what the model reads, which widens what can steer it.
Evaluating it
Retrieval and generation fail separately, so measure them separately. Build a set of thirty real questions, each with the passage that answers it and the answer you would accept. Score retrieval first: was the right passage in the top few? That number is usually the disappointing one, and no prompt fixes it. Then score the answer against the reference, by a reader or a blind comparison (Evidence and evaluation). Rerun both every time the corpus, the chunking or the model changes.
What to do with this
Take ten questions people ask about your command's instructions, and for each write down which paragraph answers it. That is your test set. Next, what to do when better documents are not the answer and the model itself has to change: Fine-tuning, LoRA and distillation.
Vendor facts checked ; re-verify before relying on them.