This question came from LT David Robinson, USN. He already knows what a language model is trained to do: put probability on the next token. He is asking how that becomes a model that restated his orbital kinematics problem as a symmetric-group calculation, an approach he could not find anywhere in the literature. The answer is that the training objective names what the model was graded on. What got built to satisfy it is a deep, composable machine that runs steps on the meaning in front of it.
The short answer
Predicting the next token well across everything anyone has written cannot be done by memorizing. The way to score is to build structure that tracks who is being discussed, what has been assumed, and what kind of problem is being posed. That structure composes, down through the layers of one run and forward across the tokens the model writes and reads back. A chain of steps where each depends on the last is what reasoning means.
The jump David saw is composition. Restating a problem in its symmetry group is a step that appears in hundreds of contexts, and attention brings it to a prompt that never named it. The last third of this chapter is what we do not know.
Four words we keep misusing
I use these four words loosely, and so does most of the industry. David's question cannot be answered until they are separated.
Prediction is the narrowest: one position's worth of output, a score for every vocabulary entry, turned into shares. Inference is the full run that produces it: every layer, every head, once. Both are fully understood.
Reasoning is the one that matters, and I will use it in a checkable sense: a chain of steps in which later steps depend on the results of earlier ones. That happens in two places: down through the layers within one run, and across tokens, where each token written becomes part of the input the next run reads.
Thinking is the everyday word, and I will keep using it for the whole loop of generating, reading back and revising, because that is what it looks like from outside. What I will not claim is that it is the process that happens in your head; that is an open question, and I say later what would count as evidence. Understanding appears only in scare quotes.
What "predict the next token" forces the network to build
Start with the grading scheme, because everything follows from it. The model produces its distribution for the next token and is scored on the probability it put on the token that came next:
$$L = -\ln p(t_{\text{next}} \mid \text{context})$$In words: the loss $L$ is the negative natural logarithm of the probability the model gave to the token that actually appeared. Put nearly all the probability there and the loss is near zero; put a tenth of it there and the loss is about 2.3. Every weight is shaped by nothing but this.
Now ask what a network must contain to make that number small on arbitrary text. To predict the closing bracket in a page of code, something is counting the open ones. To predict "she" three sentences after a name, something is tracking who is under discussion. Neither is a lookup. Both are structure that prediction pressure forced into existence, which is the point the first question makes in "Number generator" says nothing about what was built.
The best-measured example is the induction head. Olsson and colleagues described it precisely. The circuit looks back for a previous instance of the current token, finds what came after it last time, and predicts that same continuation: the pattern A, B, … A, then B (In-context Learning and Induction Heads, Transformer Circuits Thread, 2022). They analyzed 34 transformers across training with more than 50,000 attention-head ablations, meaning heads switched off one at a time to see what broke, and showed the circuits form in an abrupt phase change, at the moment in-context learning jumps.
The copy head, on three tokens
Take three positions holding A, B, A, each able to read earlier positions and not later ones. The third row matters: the position holding A for the second time.
Position three puts 0.8 of its attention on position two, 0.1 on position one and 0.1 on itself. Position two held B, so the value mixed into position three is mostly B's, and the prediction is B. Building this takes two layers: a head in the first tags each position with what came immediately before it, and a head in the second uses those tags to find the earlier A.
In words: a repeat in the context is detected by one layer and exploited by the next, and the attention row carries the answer across. Nothing was retrieved from a database; the pattern was in the prompt, and a circuit read it.
Two layers is small, but the shape is the shape of everything the model does: one layer computes, the next uses the result. And it scales. Wang and colleagues traced a circuit in GPT-2 small that resolves indirect objects, accounting for 26 attention heads in 7 classes (Interpretability in the Wild, 2022). Li and colleagues trained a network on nothing but lists of Othello moves, with no board, no rules and no images, and found a representation of the board inside it, which they could edit to change the moves it proposed (Emergent World Representations, ICLR 2023).
Connections across the weights: directions, and too many of them
David's instinct was right: embeddings must have something to do with this; perhaps there is cosine similarity between concepts. Inside the model is a running workspace, the residual stream: one vector per position that every layer reads and adds to. Concepts live in it as directions rather than as slots or neurons. One such direction is a feature: "this text is about a ship" is a feature, and so is "the thing being discussed is plural". The angle between two of them is a computable quantity:
$$\cos\theta = \frac{\mathbf{v} \cdot \mathbf{u}}{\|\mathbf{v}\|\,\|\mathbf{u}\|}$$In words: multiply the two vectors component by component, add the results, and divide by both lengths. The answer runs from 1 (the same direction) through 0 (at right angles, unrelated) to −1 (opposite). This is the same cosine similarity that ranks search results, applied to a different object: concept directions inside the model rather than passages in an index.
Take the left panel. "Ship" is $[1, 0]$ and "submarine" is $[0.8, 0.6]$; both have length 1, so their cosine is the dot product, $1 \times 0.8 + 0 \times 0.6 = 0.8$. They are related without being identical. "Tuesday" at $[-0.6, 0.8]$ scores $-0.6$ against ship and $0.8 \times (-0.6) + 0.6 \times 0.8 = 0$ against submarine, a right angle. That is the arithmetic David was reaching for, and the model runs the dot-product half of it continuously. Attention scores and read-outs are dot products between directions like these.
Now the hard part. A model has far more features to represent than dimensions to put them in, so it packs them at small angles instead of right angles. Elhage and colleagues named this superposition (Toy Models of Superposition, 2022). High-dimensional space is unusually generous about it, for reasons The Geometry of Models works through. The price is that no single neuron means one thing.
Attention moves these directions around. The same group put it plainly: the fundamental action of attention heads is moving information, reading from the residual stream at one position and writing it into another (A Mathematical Framework for Transformer Circuits, 2021). So "tying concepts across regions of the model", as David put it, is a read here and a write there, thousands of times per token. One correction: the embedding table at the front is where this starts, not where it lives. Three layers in, the vector at a position is that word as it is being used here.
Depth is composition: what many layers can do that one cannot
Every layer reads the workspace and adds to it:
$$\mathbf{x}_{\ell+1} = \mathbf{x}_{\ell} + f_{\ell}(\mathbf{x}_{\ell})$$In words: the workspace after layer $\ell$ is the workspace before it, plus whatever that layer computed. The layer does not replace the running state; it contributes to it. So layer 40 can read something layer 3 wrote, and build on it.
That "and build on it" is the whole of depth. A wider layer computes more things at once; a deeper network computes them in sequence, each step using the last one's result.
Llama 2's 7B runs 32 layers and its 70B runs 80, which is eighty chances for one computation to feed the next. That is also why the nonlinear operations between layers matter: without a bend between them, stacked linear layers collapse into one, and eighty steps buy nothing over one.
Reasoning out loud: the model's own output as scratch memory
Depth inside a pass is bounded. The second chain is bounded only by the context window, and it is the one you can see. When the model writes a token, that token joins the context and the next run reads it. Its own output is working memory:
$$p(t_1, t_2, \ldots, t_k \mid c) = \prod_{i=1}^{k} p(t_i \mid c, t_1, \ldots, t_{i-1})$$In words: the probability of a whole answer, given the prompt $c$, is the product of the probability of each token given the prompt and every token already written. That conditioning bar is the entire trick. Each step is computed with the previous steps in hand.
Ask a model how far a ship travels at 12 knots for three hours and then 8 knots for two more. Answering in one token means doing 12 × 3, then 8 × 2, then the addition, with only one pass of depth to hold the intermediate results and nowhere to put them. Let it write "12 × 3 = 36", then "8 × 2 = 16", then "36 + 16 = 52", and each step is a small calculation with its inputs in the context. That is chain of thought, and Wei and colleagues measured its worth: on the GSM8K math benchmark a 540-billion-parameter model scored 17.9% with ordinary prompting and 56.9% when shown eight examples that wrote their steps out (Chain-of-Thought Prompting, 2022).
The current generation does not wait to be asked, because the training changed. These models are finished with reinforcement learning: no next-token grading, just a reward — one number — for a whole answer, here one for a verifiably correct final answer. OpenAI's o1 system card states that the series "is trained with large-scale reinforcement learning to reason using chain of thought" (2024). DeepSeek showed the same in the open, in Nature: reasoning can be incentivized by pure reinforcement learning with no human-written traces, and the training produces self-reflection, verification and strategy-switching on its own (DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, Nature 645, 2025). Their pure-RL model's average score on the 2024 AIME set rose from 15.6% to 77.9% across training. Nobody scripted those behaviors.
David's orbit problem, unpacked
David asked for help with orbital kinematics and the model proposed recasting the problem as symmetric-group operations, dropping the kinematics entirely, an approach his literature review did not turn up anywhere. How?
Composition, as far as I can reconstruct it. Mechanics is saturated with symmetry: Noether's theorem tying conservation laws to symmetries, rotation groups for rigid bodies, Lie groups in orbital work. The training text is full of it. It also holds the step: "this is hard in its own coordinates; find the group that acts on it and work there instead", demonstrated in hundreds of contexts, and patterns that recur across contexts are what prediction pressure builds structure for. When David's prompt arrived, three things plausibly happened, inside the pass and across the tokens it wrote. The problem's structure was represented, the "symmetry applies here" direction was activated by that structure rather than by the words, and attention pulled the step across to where the answer was being written. If so, that is a chain of dependent steps, which under section two's definition is reasoning.
Two things follow, pointing opposite ways. First: the approach not appearing in the literature does not settle whether it was invented. A combination can be new for your problem and ordinary in its parts. The step is old, the domain is old, and the pairing was missing. That describes most human research insight too.
Second, and this is the part I would tell any officer using these tools: a model that can compose a move onto your problem can also compose one that looks like it fits and does not. There is no separate channel for "and I checked this": recall is generation, and so is invention. The right response is what David did: check the literature, then the mathematics.
What more parameters actually buy
This one matters, because the intuitive guess is wrong in a way that quietly poisons everything else. More parameters do not buy a bigger vocabulary; they barely touch it.
Two models, one vocabulary
Llama 2 ships at several sizes, all using the same tokenizer with a vocabulary of 32,000 entries (Touvron et al., Llama 2, 2023). The released 7B configuration is 32 layers at a width of 4,096; the 70B is 80 layers at a width of 8,192. So the input lookup table is:
$$32{,}000 \times 4{,}096 = 131{,}072{,}000 \qquad 32{,}000 \times 8{,}192 = 262{,}144{,}000$$Call those 131 million and 262 million numbers: 131 / 7,000 = 1.9% of the smaller model's parameters, and 262 / 70,000 = 0.4% of the larger's. The table doubled in width because the model got wider; its row count, the vocabulary, did not move at all.
In words: making the model ten times larger left the vocabulary where it was and cut the lookup table's share of the machine by a factor of five. Ninety-eight percent of the small model, and more than ninety-nine percent of the large one, is layers rather than word list.
So what did the extra 63 billion parameters buy? Room for more features, held with less crowding. More serial depth: 80 "and then"s in one pass instead of 32. And more heads per layer, each free to specialize. That is what I mean when I say parameters buy the ability to hold longer ideas: a longer idea is a longer chain of composed steps.
The growth is orderly. Kaplan and colleagues found the loss falls as a power law in model size, data and compute, over more than seven orders of magnitude (Scaling Laws for Neural Language Models, 2020). A power law here means the loss is proportional to size raised to a fixed negative exponent, so every tenfold increase in size cuts the loss by the same factor. Hoffmann and colleagues sharpened that to say size and training tokens should scale equally for a fixed compute budget (Training Compute-Optimal Large Language Models, 2022). What is not orderly is which abilities show up when: Wei and colleagues argued some are absent in smaller models and present in larger ones (Emergent Abilities, TMLR 2022), and Schaeffer, Miranda and Koyejo replied that the sharp jumps come largely from the choice of metric (Are Emergent Abilities a Mirage?, NeurIPS 2023). The loss curve is predictable; the map from loss to capability is not.
What is spooky, and what is not
Parts of this are spooky, though not the parts most people point at.
The recipe is fully understood: nobody is confused about the objective, the architecture or the arithmetic, and every operation in a forward pass can be written down and checked. What is only partly understood is the computation those weights ended up performing. The word "partly" does real work there, in both directions.
The largest recent step is on the "partly" side. Templeton and colleagues trained sparse autoencoders on Claude 3 Sonnet, and pulled out about 34 million features, including features for abstract things rather than surface words (Scaling Monosemanticity, 2024). A sparse autoencoder is a second, small network trained to rewrite the residual stream as a short list of named directions.
So what is spooky? Nobody predicted this from the objective: "minimize the surprise on the next token" does not sound like it should produce a machine that proposes group-theoretic reformulations of orbital problems. And the learned solution is alien in form: millions of overlapping directions in a space nobody designed. We built the loss and the architecture; the computation inside emerged, and we are reverse-engineering our own artifact.
Where the field is going
Four threads.
Mechanistic interpretability reverse-engineers the learned computation into readable pieces. Its frontier is circuit tracing: Ameisen, Lindsey, Pearce and colleagues build attribution graphs that trace the internal features behind one particular answer, then test each map by perturbing a feature to see whether the model changes as predicted (Circuit Tracing, Transformer Circuits Thread, 2025). If that scales, much of what is currently spooky becomes ordinary engineering.
Test-time compute is the shift from "make the model bigger" to "let it think for longer on this problem". The o1 and DeepSeek-R1 work above are its first shipped products, in 2024 and 2025. It trades training cost for inference cost and puts more of the reasoning where you can see it, in tokens on the page.
Tool use puts the model in a loop with things that can check: a calculator, a compiler, a search index, a simulator. A model that can propose a symmetric-group reformulation and hand it to a computer algebra system such as SymPy, or a proof checker such as Lean, is a different instrument from one that can only propose.
The open question stays a question. Is this the same kind of thinking people do? Nobody knows. My own instinct is that these models connect ideas through language in something like the way minds do. That is a hypothesis and not a finding. What would move it toward one is specific, unglamorous work: showing that the same circuit shapes recur across architectures trained on different data, that features inside a model line up with measurable structure in how people use language, and that the model's errors fall where human errors fall. Until some of that lands, "understanding" stays in the scare quotes section two put it in.
See it move
The widget below has two tabs. The first lets you type a short repeated sequence and watch the copy head's attention row light on the position that followed the earlier match. The second switches the intermediate token in a two-step problem on and off, so the answer arrives only when the working is allowed onto the page.
What you now know
- Predicting the next token is the grading scheme and not the mechanism: scoring well on arbitrary text forced the network to build structure that tracks, composes and infers.
- Reasoning, a chain of steps where later steps depend on earlier ones, happens in two places: down through the layers in one forward pass, and across the tokens the model writes and reads back.
- Concepts live inside the model as directions in a running workspace, the cosine between two of them is a real quantity, and attention carries a direction from one position to another.
- There are far more concepts than dimensions, so features sit at small angles rather than right angles, which is why no neuron means one thing.
- More parameters buy room for features, depth for longer chains and more attention heads; they do not buy a bigger vocabulary, which is fixed before training starts.
- The recipe is fully understood and the learned computation is partly understood, so "nobody knows how it works" and "we can read the weights" are both wrong.
- Whether any of this is the same kind of thinking a person does is an open question, and this chapter names what would count as evidence rather than settling it.
Where we're headed
Two threads hang out of this one. The first runs back to the first question: all of this sits on a machine that is deterministic right up to the sampler, and the chain of dependent steps I have called reasoning is a chain of rolls of a loaded die, which is why the same model reasons well on Tuesday and is confidently wrong on Wednesday. The second runs sideways to the embeddings question, which David also asked: that chapter is the exact lookup at the front door, and this one is what happens to those vectors once they are inside. Read together, they turn "it is all just embeddings and cosine similarity" into something more specific: directions, moved by attention, composed by depth.