AI Tools Field GuideWhat an LLM is (and isn't)
The model behind the chat box predicts the next small piece of text, over and over, which is why it invents facts, forgets your last conversation, and gets arithmetic wrong.
What the model does
A large language model is a program trained on an enormous amount of text to do one thing: given a passage, predict the next small piece of it. That piece is a token, roughly three quarters of a word.
- Read the passage
- Predict one token
- Append it
- Repeat
Three hundred words of reply is four hundred small predictions in a row.
That sounds too simple to produce a memo. It works because of scale. To predict text well across everything ever written, the model absorbed grammar, the shape of a brief, the look of a citation, and a lot of general knowledge.
All of that sits as patterns in billions of learned numbers, the parameters. There is no database of facts inside it that you can open and correct. The chat product is a wrapper around that predictor; The harness is the product covers the wrapper.
Three consequences you will feel on day one
| It is… | So… | What you do |
|---|---|---|
| Memoryless between calls. Every reply is generated from scratch from whatever text is sent this time. | "Memory" features are the harness re-sending saved text. Nothing persists inside the model. | Put durable material in a project or a file, and restate what matters. |
| Built to produce a plausible answer. It produces a confident answer whether or not one exists. Training graded every word it predicted against the word that came next, and nothing in that grading pays for silence. A blank earns nothing, while a guess that lands earns full marks, so guessing is the habit it learned. | This is the root of hallucination. The most likely continuation of a question is an answer, not "I don't know." | Give it the source. Tell it "I don't know" is fine. Verify. |
| A language engine, not a calculator or a database. | Arithmetic, dates, counts, and citations are where it slips, because it predicts the shape of a right answer rather than computing one. | Route those to a tool or check them by hand. |
Each row explains a failure you would otherwise blame on yourself.
| The fact | What it looks like at your desk |
|---|---|
| Memoryless | You open a new chat and say "as we discussed yesterday." There is no yesterday. If the product seems to remember, the harness saved some text and put it back into today's request. Only what was saved comes back, and you did not choose what was saved. A project file you wrote is a memory you control. |
| Plausible | You ask which instruction governs a topic and get a number in the right format with a title that sounds right. It may exist. Real answers to that question contain instruction numbers, so the model produces one. Nothing checks it. The same goes for quotations, statistics and references, and no later version removes this, because it follows from how the method works. |
| A language engine | You ask how many days between two dates and get a fluent answer. It is predicting the digits that usually follow such a question, not counting days. Products bolt on a calculator or a code tool for this reason. Without one, treat every number it produces as a draft. |
What it is good at
The same training that makes it guess at facts makes it strong at transforming text you supply: restructuring, condensing, changing register, comparing versions, drafting in a form you show it. In all of those it is rearranging material that is in front of it.
When a fact does come out of the model, it comes out by the same weighted draw as every other word. That is why supplying the material beats asking it to remember. How the model decides covers that draw in plain words.
| Strong when the material is in the request | Weak when it must supply the material itself |
|---|---|
| Summarize this instruction into what changed | What does the current instruction say? |
| Rewrite these notes as a memo in this format | Write a memo on our leave policy |
| Compare these two drafts and list the differences | Which version is the latest? |
| Turn this roster into a table with these columns | How many people are in the division? |
| Draft award citations from these bios and this example | Draft an award citation for Petty Officer Smith |
The right-hand column has one pattern: the model is asked to know something about your command, your dates or the state of the world. It cannot. What it produces there is fluent guesswork.
A staff example
| Asked from memory | Asked from the attachment |
|---|---|
| "What are the reporting requirements after a mishap?" | "Using only the attached instruction, list the reporting requirements after a mishap and quote the paragraph for each." |
| A clean numbered list of five with an instruction cited. Two items are right, one is from an older version, one belongs to another service, and the instruction number is a digit off. Nothing looks wrong. | A list you can check line by line against the quoted paragraphs in a minute. |
What to do with this
Next time a reply surprises you, ask which of the three it was: forgot because nothing was re-sent, invented because nothing was supplied, or miscounted because nobody gave it a calculator. Then read Trust and limits, which turns those facts into a method.