Hi, I'm Sean Egan, a Navy information warfare officer and research scientist. This is the foundations book of the set, and it makes one argument the whole way through. A language model is a geometry, and everything we use it for, from similarity and retrieval to reranking and surprise, is a measurement taken inside that space.
Learn the space first and the tricks stop being tricks. Start at Chapter 1; it assumes arithmetic and nothing else. If you have read the LLM book and want the part nobody explains, go straight to Chapter 3.
What you need first
Arithmetic, and a willingness to look at an equation for a few seconds. That is the floor. Every symbol is defined the first time it appears in a chapter, even if an earlier chapter already defined it, and every worked example uses numbers small enough to check on the back of an envelope.
If you have read the other books in this set, Part I will feel familiar for about a page and then go somewhere they do not. Chapters 3 and 4 are the reason this book exists: the behavior of many-dimensional space, and how to read a picture of it without fooling yourself. Neither is covered anywhere else in the family.
Part I
Space Itself
No machine learning yet. Four chapters on the space every model lives in, built out of arrows you can draw on paper, plus one chapter that breaks the drawing on purpose.
-
1Numbers as ArrowsA place, a heading, and one number for how much two of them agree12 min
2Distance and AngleTwo rulers for the same pair of points, and how to choose between them11 min
3The Weirdness of Many DimensionsConcentration of measure: where flat-land intuition stops being true12 min
4Projections and ShadowsChoosing the flashlight angle, and what a two-dimensional picture is allowed to prove10 min
Part II
Meaning as Position
Where a word's position comes from, what arithmetic on positions really proves, how a map is trained into shape, how you search one, and then ten thousand real embeddings you can walk yourself.
-
5The Embedding IdeaWhere a word's coordinates come from, counted by hand14 min
6Arithmetic with MeaningOffsets as relations, and the famous example taken apart14 min
7Learning the MapContrastive training: which pairs pull and which push13 min
8Using the MapNearest neighbors, centroids, clusters, drift, and the recall dial13 min
9Take a Real Dataset for a WalkTen thousand real embeddings, in your browser, with the caveats live14 min
Part III
Geometry Inside the Model
What happens to a position once a language model gets hold of it: context moves it, every layer adds to it, the directions it moves along are what the model has learned, and the last step reads it as a probability and a number of bits.
-
10Context Moves the PointThe same token, two sentences, two places15 min
11The Residual StreamOne vector, read and written by every layer16 min
12Features as DirectionsSuperposition, probes, and the knob Part II handed you16 min
13The Surprise SurfaceFrom one point to four probabilities, and the bill in bits17 min
Part IV
Instruments and Invention
What one vector can and cannot carry, the whole instrument kit running as one pipeline with the numbers printed, and three experiments you can run yourself on the open questions.
What is here
Part I is the instrument kit with no machine learning in it, Part II puts meaning into the space, Part III follows a position through a working model, and Part IV is the instruments and what nobody has settled yet.
- Part I · Space Itself Numbers as arrows, distance and angle, the weirdness of many dimensions, projections and shadows. Four chapters
- Part II · Meaning as Position Where a word's position comes from: co-occurrence counts worked by hand, arithmetic on meaning and where it is folklore, contrastive training read as forces on points, searching a map of millions, and ten thousand real Wikipedia embeddings you can walk yourself. Five chapters
- Part III · Geometry Inside the Model Context moving a point, the residual stream as one journey, features as directions, and the probability surface read as surprise. Four chapters
- Part IV · Instruments and Invention What one point cannot carry and why rerankers read both texts together, the instruments as they are used in a working system, and the open questions at the frontier with the experiments you could run yourself. Three chapters
The foundations book of five: this book · AI/ML from the Ground Up · Training a Model · How LLMs Actually Work · Diffusion (images).