You have already seen a thousand of these
A two-dimensional plot of embeddings is a shadow of a space with hundreds of dimensions, and a shadow has rules about what it can and cannot establish.
Open any paper about embeddings and somewhere in it there is a scatter plot: a few hundred colored dots on a flat pair of axes, with clusters labeled "sports," "medicine," "recipes." It is the picture everyone reaches for, and it is useful. The previous chapter told us those dots live in 1000 dimensions, so the plot is not the space.
A shadow is two readings
We built the parts for this in Chapter 1 and have not used them since. A unit vector is a ruler, and dotting a point with it hands back one number: how far the point reaches along that direction.
To put a point on a page you need two numbers. So choose two ruler directions, take two readings, and plot the results. That is the entire operation.
$$\text{across} = \mathbf{x} \cdot \hat{\mathbf{a}}, \qquad \text{up} = \mathbf{x} \cdot \hat{\mathbf{b}}$$In words: the horizontal position of a dot is the point's reading along the first chosen direction, and the vertical position is its reading along the second. Nothing else about the point survives onto the page.
Pick the two rulers at right angles to each other, so the two readings do not double-count the same information. The result is called an orthogonal projection: the shadow a point casts when a light shines straight down onto the plane those two rulers define.
The three panels above are one cloud of points, photographed from three directions. Nothing about the data differs between them. Those are true projections of the same list of points onto three different planes. Turn the flashlight and groups that were piled on top of one another come apart, or the reverse. A shadow always loses something, and which something it loses is a choice somebody made.
The question to ask of a plot is which of the true things it chose to keep.
Choosing the angle on purpose
If the angle is a choice, make it deliberately. The standard way is PCA, principal component analysis, and the idea is one sentence: point the flashlight so the shadow is as spread out as possible.
A direction along which the data barely varies carries almost nothing to distinguish one point from another, so losing it costs little. A direction along which the data varies a lot is doing work worth keeping. So rank every possible direction by how much the data spreads along it, and keep the top two.
Two housekeeping steps come first, and both matter. Center the data by subtracting the average point, so the readings describe displacement from the middle of the cloud instead of from an arbitrary origin. And use unit ruler directions, so a reading is a coordinate and not a coordinate multiplied by some leftover scale factor.
Finding the best angle by hand
Four points, already centered on the origin: $[3, 4]$, $[-3, -4]$, $[-0.8, 0.6]$ and $[0.8, -0.6]$. Try three candidate rulers and see which spreads them most. Spread here is the average of the squared readings, the ordinary variance.
Ruler 1, straight across, $[1, 0]$. The four readings are 3, −3, −0.8 and 0.8. Squared: 9, 9, 0.64, 0.64. Average: $19.28 \div 4 = 4.82$.
Ruler 2, straight up, $[0, 1]$. Readings 4, −4, 0.6, −0.6. Squared: 16, 16, 0.36, 0.36. Average: $32.72 \div 4 = 8.18$.
Ruler 3, the tilted one, $[0.6, 0.8]$, our unit vector from Chapter 1. First point: $3 \times 0.6 + 4 \times 0.8 = 1.8 + 3.2 = 5$. Second: −5. Third: $-0.48 + 0.48 = 0$. Fourth: 0. Squared: 25, 25, 0, 0. Average: $50 \div 4 = 12.5$.
In words: the tilted ruler spreads these points more than either of the two we were handed, 12.5 against 8.18 and 4.82. It is the first principal component of this little cloud, and neither of the axes the data arrived with was it.
Now the second component. It must be at right angles to the first, which in two dimensions leaves one choice: $[-0.8, 0.6]$. Its readings are 0, 0, 1, −1, so its variance is $2 \div 4 = 0.5$.
Add the two up: $12.5 + 0.5 = 13.0$. Now add up the two we started with: $4.82 + 8.18 = 13.0$. The same total. Turning the rulers moved the spread around between them but never created or destroyed any, and that conservation is what makes the next number meaningful.
$$\text{share kept} = \frac{12.5}{13.0} \approx 0.96$$In words: the first direction alone accounts for about 96% of all the spread in this cloud. Keep it and throw the other away, and you have lost 4% of what distinguished these four points.
That percentage is the variance explained, the most important number on any embedding plot and the one most often left off. A two-dimensional shadow of a 1000-dimensional space frequently keeps less than 15% of the spread. The picture may still be worth looking at. But you are looking at a seventh of the evidence, and nobody has told you which seventh.
What a shadow is allowed to prove
Two rules govern what a plot can establish, and they point in opposite directions. Memorize both.
When a point is projected, its journey splits into two perpendicular pieces: the part that landed on the page and the part that was lost. Those two pieces and the original form a right triangle, so Pythagoras applies, which gives a hard guarantee.
$$\|\mathbf{x}\|^2 = \|\text{shadow}\|^2 + \|\text{lost}\|^2$$In words: the original length squared equals the shadow's length squared plus the lost part's length squared. Because the lost piece can never be negative, the shadow is never longer than the original.
Apply that to the gap between two points and the two rules fall out:
- Distance in the shadow is never more than the true distance. So if two dots look far apart on the page, they are far apart in the full space. A shadow cannot manufacture a separation that was not there.
- Distance in the shadow can be much less than the true distance. So two dots sitting on top of each other prove nothing. They may be neighbors, or they may be on opposite sides of the space with their difference pointing in a direction the projection discarded.
See it move
Turn the flashlight yourself. The cloud below has three real groups in three dimensions. Rotate the viewing direction and watch the groups merge and separate in the shadow beneath, with the share of spread kept reported live.
Where this shows up
- Every embedding visualization you will ever be shown. The first two questions are now automatic: what share of the spread does this keep, and is this a projection or a neighbor-preserving rearrangement? If the caption answers neither, the picture is decoration.
- Dimensionality reduction as a preprocessing step. Cutting 1000 dimensions to 100 before clustering or search is the same operation with more rulers kept. The AI/ML book's classical toolbox shows where it sits in a working pipeline.
- Reading a model's internals. When researchers look for structure inside a network's activations, they are projecting onto a handful of chosen directions and reading the shadow. The guarantees in this chapter are the guarantees on their conclusions.
- Compression and storage. Keeping the top directions and discarding the rest is how a vector store shrinks its index, and the variance-explained figure states what that saving cost.
What you now know
- A two-dimensional plot of embeddings is a shadow: two readings taken along two chosen perpendicular directions, with everything else discarded.
- Principal component analysis chooses those directions by ranking every direction by how much the data spreads along it and keeping the top ones, after centering the cloud.
- Turning the rulers moves spread between them without changing the total, which is what makes "share of the spread kept" a meaningful number.
- A projection can only shrink distances, so separation on the page is real evidence while overlap on the page is no evidence at all.
- Neighbor-preserving pictures such as t-SNE and UMAP are not projections, and neither of those rules applies to them.
Where we're headed
That completes Part I. You have the whole instrument kit: a space, two rulers, an account of what many dimensions do to both, and a way of looking at the result without fooling yourself. None of it mentioned machine learning. These are facts about space that machine learning depends on.
Part II, Meaning as Position, starts at The Embedding Idea. It puts meaning into the space: where a word's coordinates come from, counted by hand from ten dull sentences; what "king minus man plus woman" really proves; how training pulls and pushes points into position; how you search a map of millions; and then ten thousand real Wikipedia embeddings you can walk through in your browser. If you would rather take the next rung of the ladder instead, that is AI/ML from the Ground Up, which takes the space you have just built and starts fitting things in it.