Part I · Ch. 3 — The Weirdness of Many Dimensions

Part I · Chapter 3 of 16

The Weirdness of Many Dimensions

Concentration of measure: where flat-land intuition stops being true


The page has been lying to you

Space in many dimensions behaves in ways that have no flat-page analog, and those differences are the reason embeddings work.

Every picture so far has been drawn flat, on a page, because a page is something you can check with a pencil. The same flatness is why most people who use these tools carry an intuition that quietly fails them. A real model does not work in two dimensions. It works in 384, or 1024, or 4096.

The usual reassurance is that high dimensions are just more of the same: same arithmetic, more slots. The arithmetic half is true and Chapter 1 proved it. The geometry half is false. This chapter runs one experiment four times. Draw two arrows at random, measure the angle between them, and do that twenty thousand times, first on a page, then in 10 dimensions, then 100, then 1000.

The experiment: twenty thousand random pairs

The procedure is short enough to hold in your head. Pick a direction at random, every heading equally likely. Pick a second one the same way, with no reference to the first. Measure the angle between them. Write it down. Repeat.

On a page, with only two dimensions to play in, you already know roughly what to expect. The second arrow can land anywhere on the circle, so the angle between them is a coin toss across the whole range: sometimes nearly aligned, sometimes nearly opposite, sometimes square. No angle is favored, and the first panel below shows that as a flat histogram, every angle about equally common.

Four histograms in a row, all with the same horizontal axis running from 0 to 180 degrees and a dashed vertical line marked at 90 degrees. The first, headed d equals 2, is flat and low across the whole range: every angle turns up about equally often. The second, d equals 10, is a wide mound centered on 90 degrees. The third, d equals 100, is a much narrower and taller peak on 90 degrees. The fourth, d equals 1000, is a very tall narrow spike sitting on 90 degrees with nothing elsewhere.
The same experiment, four times. 20,000 random pairs at each dimension count. As the space gains dimensions, the angle between two unrelated directions collapses onto a right angle and stays there.

The measured numbers are stranger than the picture lets on.

Angle between two random directions. 20,000 sampled pairs at each dimension count.
Dimensions $d$Average angleSpread (one standard deviation)Share within 10° of a right angle
289.8°51.9°11.5%
1090.0°19.2°39.0%
10089.9°5.8°91.6%
100090.0°1.8°100.0%

The average never moves. It sits on a right angle at every dimension count, which is the dull half of the result, since a random direction is as likely to lean one way as the other. The spread is the other half. At $d = 1000$, all twenty thousand pairs landed within ten degrees of square: 20,000 out of 20,000, and the worst pair in the whole run was 7.7 degrees off.

Two directions at right angles are called orthogonal, and their dot product is zero, meaning they agree about nothing. So the finding, in one sentence: in a space with many dimensions, two things chosen without reference to each other are almost always almost unrelated, and the more dimensions there are the more reliably this is true. This behavior has a name, concentration of measure, and it governs the spaces models live in.

Why the votes cancel

The mechanism is easier than the result. A dot product between two unit arrows walks down both lists, multiplies each pair of matching entries, and adds up $d$ separate contributions.

Each contribution is a small number as likely to come out positive as negative, because neither arrow knows anything about the other. So you are adding up $d$ random votes, half for and half against. With two votes, one side can win outright and every margin is as likely as every other, which is the flat first panel. With a thousand votes drawn the same way, the chance that they fail to mostly cancel is vanishingly small.

They do not cancel perfectly. The leftover wobble shrinks as the square root of the number of votes, which gives a rule you can check against the table above:

$$\text{spread of } \cos\theta \approx \frac{1}{\sqrt{d}}$$

In words: the typical size of the alignment score between two unrelated directions is about 1 divided by the square root of the number of dimensions. Quadruple the dimensions and the wobble halves.

Checking the rule against the measurements

At $d = 100$ the rule predicts a spread of $1 \div \sqrt{100} = 1 \div 10 = 0.1000$. The twenty thousand sampled pairs gave 0.1004.

At $d = 1000$ the rule predicts $1 \div \sqrt{1000} \approx 0.0316$. The samples gave 0.0315.

At $d = 10$: predicted $1 \div \sqrt{10} \approx 0.3162$, measured 0.3176. At $d = 2$: predicted $1 \div \sqrt{2} \approx 0.7071$, measured 0.7055.

In words: four predictions and four matches, every one inside half a percent, from a rule you can do in your head. It holds to that precision across three orders of magnitude, and it is the most useful number to carry around about a model's embedding size.

Almost everything is skin

A ball in many dimensions is all surface, with no interior worth speaking of. This is the companion result to the angle one, and the one that most offends a flat-page intuition.

Take a ball of radius 1. In two dimensions that is a disc, in three a solid sphere, in a thousand something we cannot draw but can measure. What fraction of it lies in the thin outer rind, the part between radius 0.9 and the surface?

The answer is arithmetic you can do on paper. Volume scales with the $d$-th power of the radius, so the inner ball of radius 0.9 holds a fraction $0.9^d$ of the whole, and the rind is whatever is left over.

$$\text{fraction in the outer rind} = 1 - 0.9^{\,d}$$

In words: raise nine tenths to the power of the number of dimensions to find how much of the ball is in the inner core, and subtract that from 1 to find how much is in the outer tenth of the radius.

A curve on axes labeled 'number of dimensions d' running from 1 to 100 along the bottom and 'share of the ball in its outer tenth' from 0 to 100 percent up the side. The curve starts at 10 percent at d equals 1, rises steeply through about 65 percent at d equals 10, and is flat against the 100 percent line from about d equals 60 onward. Three points are marked with their values at d equals 2, 10 and 100.
The inside empties out. By 100 dimensions, 99.997% of the ball lies in the outer tenth of its radius. There is almost no interior left.

The rind, by hand

On a page, $d = 2$: the core is $0.9^2 = 0.81$, so the rind holds $1 - 0.81 = 0.19$, about a fifth. That matches what your eye expects from a dartboard.

At $d = 10$: the core is $0.9^{10} \approx 0.349$, so the rind holds about 65%. Already most of it.

At $d = 100$: the core is $0.9^{100} \approx 0.0000266$, so the rind holds 99.997%.

At $d = 1000$ the core is smaller than one part in $10^{45}$, a number with no useful name.

In words: in the spaces models use, a ball is all surface. Pick a point inside one at random and it is sitting on the skin, and so is every other point you will ever pick.

Put that beside the angle result and you have the picture of a high-dimensional space: everything lives on a thin shell, and everything on that shell is at right angles to everything else. It is nothing like the inside of a room.

Nearest is barely nearer

In high dimensions, the nearest point and the farthest point are at almost the same distance. Entire products are built on the assumption that they are not.

Scatter a thousand points at random. Stand at a new random point and ask for the closest one and the farthest one. On a page that is a meaningful question: the closest point is practically underfoot and the farthest is across the room. Now do it in more dimensions.

Nearest and farthest of 1,000 random points, averaged over 200 query points.
Dimensions $d$Distance to nearestDistance to farthestHow much farther the farthest is
20.0161.033115× as far
100.4991.9122.9× as far
1003.3264.7720.44× as far
100012.19113.6300.12× as far

Read the last column again. On a page, the farthest point is more than a hundred times farther away than the nearest one, so "nearest" is a strong and obvious answer. At 1000 dimensions the farthest point is only 12% farther than the nearest. All thousand points sit at roughly the same distance from you, and "the nearest" is a photo finish decided in the third decimal place.

See it move

Watch (12:43): what to notice — the same experiment run four times — the flat band of angles at two dimensions pulls into a single spike at a thousand, while the average never moves off a right angle; the one-over-root-d rule predicts all four measured spreads inside half a percent; and a ball stops having an inside, which is exactly the room a model needs.

Do not take the table on trust. The widget below runs the experiment live in your browser. It draws fresh random directions, measures the angles, and builds the histogram in front of you. Move the dimension slider and watch the shape collapse.

Why this is good news

Near-orthogonality is what makes an embedding space useful. The list above reads like things going wrong; it is the opposite.

Suppose you want a space to hold a direction for every distinct idea a model might need: "formal," "in French," "is a place name," "is talking about grief." If directions had to sit at right angles to avoid interfering, a space with $d$ dimensions could hold $d$ of them, and 1024 ideas is nowhere near enough for a language.

They do not have to be exact, though. They only have to be far enough apart that reading one does not pick up much of another, and we now know what "far enough" costs. In 1000 dimensions, two directions picked at random are within about two degrees of square, so a reading taken along one of them picks up a leak of about 0.03 from the other. That leak is small enough to ignore, and it stays small for an enormous number of directions. Not 1000 of them. Millions.

So the summary of high-dimensional space is this: the space is enormous, almost all of it is at right angles to almost all the rest, and that is the room a model needs to keep a world's worth of distinctions from trampling one another.

Where this shows up

  • Choosing an embedding size. The 1-over-the-square-root-of-$d$ rule tells you what a similarity score is worth before you have run a single query. It is why moving from 384 to 1536 dimensions changes what a threshold of 0.3 means.
  • Setting a similarity threshold. Any fixed cutoff, for deduplication, for "is this relevant enough to retrieve," for "has this person's style changed," is a statement about how many standard deviations of noise you are willing to accept. Compute the noise floor first.
  • Approximate search. Index structures that trade exactness for speed are living on the photo-finish result. RAG and Vector Search shows how they are built; this chapter explains why the approximation is affordable.
  • Reading interpretability work. Steering vectors, linear probes and sparse feature dictionaries all assume you can find many nearly-independent directions in one space. That assumption is the measurement we just made.

Which leaves one problem. If almost nothing is visible in two dimensions, what are we looking at when somebody shows us a two-dimensional plot of an embedding space?

What you now know

  • Two directions drawn at random in a high-dimensional space are almost always close to a right angle, and the tightness of that result grows with the number of dimensions.
  • The spread of the alignment score between unrelated directions is about 1 divided by the square root of the dimension count, which four separate measurements confirmed to three decimal places.
  • A similarity score cannot be interpreted without knowing the dimension count, because the same number is noise in a small space and a strong signal in a large one.
  • Almost all of a high-dimensional ball's volume lies in a thin shell at its surface, so there is effectively no interior to occupy.
  • Nearest and farthest neighbors sit at nearly the same distance in high dimensions, which is why approximate search is affordable and why small model improvements are visible.
  • All of this is what lets a space of 1024 numbers carry millions of distinguishable directions, which is the mechanism behind superposition.

Where we're headed

The interesting structure lives in hundreds of dimensions and a flat page cannot hold it. And yet every paper, every blog post and every dashboard shows embeddings as a two-dimensional scatter plot. Those pictures are shadows, and a shadow is a real measurement with real rules about what it can and cannot prove. The last chapter of this part is about choosing the flashlight angle on purpose, and about what everybody else's shadows can and cannot show.