Text Becomes Numbers

A machine that has never seen a letter

Here is a fact worth sitting with. The same models that draft flawless essays and working code will, asked how many r's are in "strawberry," confidently answer two. Not because counting to three is hard — because they have never seen an r. Or an s. Or any letter at all. A large language model is, in a strict sense, a reader that has never once looked at a letter in its life.

Back in Chapter 6 we kept saying "word-piece" and promised the precise unit would arrive with its real name. This is the chapter that pays that debt. Before any of Part I's mathematics — the arrows, the dot products, the curves — can touch a single sentence, the text has to be chopped into a fixed menu of countable pieces and swapped for numbers. And the chopping is done by a small, separate program with a personality all its own, sitting in front of the model like a turnstile everything must pass through.

Here is the plan. First, why neither letters nor whole words work as the pieces — the Goldilocks problem. Second, the counting game that builds the menu, played by hand from start to finish. Third, the moment text becomes a plain list of integers. And last, the quirks this chopping quietly explains — the miscounted strawberry very much included.

The Goldilocks problem: letters, words, or something between

Let me clear away the first confusion, because it is a good one. Computers already store text as numbers — every character you type has a code number under the hood, has for decades. So "turn text into numbers" sounds like a solved problem. It isn't, and here is why: those code numbers are arbitrary labels stuck onto single characters. What the model needs is something different — a fixed menu of pieces it will later learn to attach meaning to (meaning is Chapter 8's entire job). The real design question is: what should the pieces on that menu be?

Try the two obvious answers first. Option one — single characters. The menu is tiny: every letter, digit, and punctuation mark ever typed fits in a few hundred entries. Wonderfully compact. But each piece means almost nothing on its own — the letter c tells you nothing about cats — and every sentence becomes an enormously long ribbon of nearly meaningless crumbs. Too fine.

Option two — whole words. Now each piece carries real meaning. But the menu explodes. Every name, every typo, every plural, yesterday's slang, every German compound noun, a coinage like "unhuggable" — either the menu balloons to millions of entries, or the model meets a word it has no piece for and cannot read it at all. Too coarse, and brittle right at the edges where language is most alive.

So we split the difference — the Goldilocks answer: reusable chunks that live between letters and words. Here is the word for one: token, a chunk of text the model treats as one indivisible unit. Common words each get to be a single token; rare words get built up out of a few sub-word tokens. A rule of thumb to plant now and reuse all site: in English, a token is on average about three-quarters of a word.

That leaves one question, and the next section answers it with a game: who decides where the chop-points go? Nobody does. They are discovered, by counting.

The merging game

Let me give you the rules before the name. Start with your training text chopped all the way down to single characters. Now repeat one move: find the most frequent adjacent pair of pieces anywhere in the text, and glue that pair into one new piece. Then do it again. And again — tens of thousands of times. That is the entire game. Its name, now that you have played it in your head, is byte-pair encoding (BPE), and it is the method behind the tokenizers of essentially every modern LLM.

Let's play it on numbers small enough to hold. Our entire training text — and I am making this up for teaching, it is not real English data — is three words, with counts: hug appears 10 times, pug 5 times, and hugs 5 times. Chopped to characters, our starting menu has five pieces: h, u, g, p, s.

So what did the game actually learn? Not a dictionary of words — an ordered list of merges: (1) u+g → ug, then (2) h+ug → hug. That little numbered list is the tokenizer. To chop any future text, you chop it to characters and replay the merges in order, wherever they apply.

Watch it handle words it never saw during the game. Take pugs: chop to p·u·g·s, then replay. Merge 1 (u+g → ug) applies, giving p·ug·s. Merge 2 needs an h in front of the ug and there isn't one, so it doesn't apply. Final answer: three tokens, [p][ug][s]. And hug? Both merges fire, and it collapses to a single token [hug]. There is the pattern the whole game exists to produce: frequent strings end up as one crisp token, rare strings get built from a few reusable parts. The menu quietly shapes itself to the language it was counted on — and nobody ever chose a single chop-point by hand.

One honesty note, kept to a single sentence: real tokenizers play this exact game starting from raw bytes rather than letters — a technicality that lets them handle emoji, accented letters, and every writing system on Earth with no "unreadable character" escape hatch.

Catalog numbers: IDs and the vocabulary

Give the finished menu its name. Every piece the game ever produced — the starting characters plus each merge — goes into one big list called the vocabulary. Its size earns a letter we will lean on for the rest of the site: $V$, the vocabulary size — the number of distinct tokens in the menu. Ours holds h, u, g, p, s, ug, hug, so $V = 7$.

Each token gets a token ID: its row number in that list. In our order of creation: h = 1, u = 2, g = 3, p = 4, s = 5, ug = 6, hug = 7 (these ID numbers are made up too, of course — a toy list). Now run the whole pipeline on our two test words. hugs chops to [hug][s], which reads off as IDs [7, 5]. pugs chops to [p][ug][s], which reads off as [4, 6, 5]. Text in, list of integers out. And going the other way — the model's output back into text — is the very same table, read backwards.

Our toy vocabulary has 7 entries. Real ones land around fifty thousand — one famous early model's is exactly 50,257 — and the newest models often run to one or two hundred thousand. But whatever the count, the principle never changes: every text ever written gets expressed as a sequence drawn from one fixed menu of $V$ pieces.

So hold this line carefully, because it sets up the next chapter: an ID is a catalog number, not a measurement. hug = 7 and s = 5 does not make hugging 40% more of anything than s. The numbers carry no meaning yet — none — and giving each ID a meaning turns out to be a whole separate piece of machinery.

See it happen

Everything above is much easier to feel in motion. The short video watches "strawberry" lose its letters, then plays the merging game on our exact corpus, then chops a brand-new word by replaying the merges — landing on the same numbers you just worked out by hand.

One self-check while you watch: when the vocabulary panel appears, pause and read off the ID for ug before the video reveals it. It's 6 — the first merge, so the first entry after the five starting characters.

Quirks you can feel in the chatbot

Now the hook pays off — strawberry, resolved. A typical tokenizer chops strawberry into two tokens, something like [straw][berry] (I say "something like" because the exact split varies by tokenizer). The model receives two catalog numbers and nothing else. The ten letters — and the three r's hiding among them — are not in the delivery at all. Asking the model to count the r's is asking it about objects it never received. When it happens to answer correctly, it is because it memorized a fact about the word, not because it looked inside it.

Quirk two — the invisible space. Most tokenizers glue the leading space onto the word that follows it, so " the" (with a space) and "the" (without) are two different tokens with two different IDs. This is why models occasionally do strange things at the seams of words, and why the very same word can cost a different number of tokens depending on where it sits in a sentence.

Quirk three — numbers get chopped arbitrarily. A string like 12345 might come through as [123][45] — two tokens whose boundaries have nothing whatever to do with place value. Some of the famous arithmetic clumsiness of LLMs starts right here, at the chopping board, before any "thinking" has happened at all.

Quirk four — tokenization is uneven across languages. A sentence in English might cost 20 tokens while its faithful translation costs 40, because the merging game was played mostly on English-heavy text — so English got the efficient one-token words and other languages got left assembling theirs from scraps. Same meaning, double the pieces.

And a practical aside, because it explains a number you've seen quoted. This is why prices, context limits, and speed are all measured in tokens — the token, not the word, is the model's natural unit of text. So when a provider says a model handles 200,000 tokens of context, you can now translate on the spot: that's roughly 150,000 English words (about three-quarters of a word per token, on average).

Why the LLM cares

Here is where this small program sits in the whole machine: the tokenizer is the model's entire sensory apparatus. Everything you have ever typed to a chatbot became a list of token IDs before the model perceived anything at all; everything it ever said back was a list of IDs turned into text at the very last instant. There is no side channel — no letters, no sounds, no images of words slipping in around the side. Tokens in, tokens out.

That lets us sharpen Chapter 6's star equation to its final precision. We display it once more:

In words: the probability of each possible next token, given the tokens so far.

The "word-pieces" from Chapter 6 were tokens all along, and the bars of that distribution are one per vocabulary entry — all $V$ of them, tens of thousands of bars, rebuilt from scratch before every single choice the model makes.

That same $V$ has one more job waiting a couple of chapters out (a name-drop only, no mechanics yet): it fixes the size of the model's final answer. The model's last act before choosing a word will be to produce one score per vocabulary token — $V$ scores, every time.

Let me close by pointing straight at the gap this chapter left open on purpose. We turned text into numbers — but they are the wrong kind of numbers. Catalog numbers, meaning-blind by design. Cat and dog get IDs no more related than two random library cards. All of Part I built tools for arrows, angles, and similarity, and so far we've had no way to point them at language. Next chapter, every ID finally gets traded for an arrow — and the whole toolkit comes out of the box.

Where we're headed

So text is now numbers — but deliberately dumb ones. Token IDs are library cards: they tell you which piece showed up and absolutely nothing about what it means. Cat and dog land on numbers no more related than two strangers' phone numbers. And this is exactly where Part I stops being warm-up and starts being the point. In the next chapter, every token ID gets traded for a vector — an arrow, straight out of Chapter 1 — and suddenly the dot product from Chapter 2 can measure whether two words mean similar things. Meaning is about to become geometry.