Part II · Ch. 7 — Text Becomes Numbers

Part II · Chapter 7 of 21

Text Becomes Numbers

Tokens: chopping language into countable pieces

You’ll need:the model's one move: build p(next | context) over word-pieces, sample, repeat

A machine that reads pieces

Why can a fluent model stumble when counting letters in “strawberry”? Its input arrives as numbered text pieces, not a row of separately labeled letters. We'll build the program that makes those pieces and follow a word through it.

The Goldilocks problem: letters, words, or something between

Computers already encode text as numbers. The design question is which units the language model should process. A token may be a word, part of a word, punctuation, or a byte.

The word unbelievably chopped three ways. Top row, labeled letters: twelve separate character boxes u n b e l i e v a b l y, annotated 12 pieces, none means anything alone. Middle row, labeled whole words: one wide box holding unbelievably, annotated 1 piece but the menu needs millions of entries. Bottom row, labeled tokens: three boxes un, believ, ably, annotated 3 reusable pieces, small menu, pieces still mean something.
Reusable pieces balance length and vocabulary.
Choose the size of a text piece
UnitAdvantageCost
CharacterReusable small pieces.Long sequences; the model must combine many pieces. All Unicode characters require a large inventory.
Whole wordShorter sequences of familiar units.An enormous menu; new names and spellings need a fallback.
SubwordCommon strings become one piece; rare ones share parts.Boundaries need not match syllables.

A tokenizer applies the chosen segmentation rules. About three-quarters of an English word per token is a rough guide, not a rule for every language or passage.

The merging game

Byte-pair encoding, or BPE, learns reusable pieces from a corpus. The procedure runs like this:

  1. Start with small base pieces.
  2. Count adjacent pairs, weighted by their occurrences.
  3. Merge the most frequent pair and recount.
  4. Stop at the chosen vocabulary size; retain the merge order.

Our invented corpus has hug ten times, pug five times, hugs five times. Start with h, u, g, p, s.

Two rounds count adjacent pairs: u+g wins with 20, then h+ug wins with 15; the final corpus retains all word counts.
Merge ug first, then hug.

Worked example: Round 1

Count each adjacent pair across all three words, weighting by how often the word appears:

  • h+u shows up in hug (10) and hugs (5) = 15
  • u+g shows up in all three: hug (10), pug (5), hugs (5) = 20
  • p+u shows up in pug only = 5
  • g+s shows up in hugs only = 5

The winner is u+g with 20. Glue it: a new piece ug joins the menu, and every u-g in the corpus fuses at once. The words are now h·ug (×10), p·ug (×5), and h·ug·s (×5).

Worked example: Round 2

Recount the pairs, now on the new pieces. This is the last round I'll count in full; after this we summarize.

  • h+ug: h·ug (10) and h·ug·s (5) = 15
  • p+ug: p·ug (5) = 5
  • ug+s: h·ug·s (5) = 5

Winner: h+ug with 15. Glue it, and the piece hug exists. The corpus is now hug (×10), p·ug (×5), and hug·s (×5). (Round 3 would be a 5-vs-5 tie between p+ug and hug+s; real tokenizers break ties by a fixed rule and keep going for tens of thousands of rounds. We stop here.)

Replay the learned merges on new text
Stagepugshugs
Startp · u · g · sh · u · g · s
u + g → ugp · ug · sh · ug · s
h + ug → hugp · ug · shug · s

For this toy tokenizer, the ordered merge list determines the split. Production tokenizers also specify preprocessing, special tokens, and tie-breaking. BPE is common; other tokenization methods exist.

Byte-level BPE starts from bytes. That provides a fallback for emoji, accented letters, and unfamiliar scripts without needing a whole-word entry for each.

Catalog numbers: IDs and the vocabulary

The vocabulary contains base pieces and learned pieces. Its size is V. A token ID identifies one entry.

Left: the vocabulary as a seven-row table titled the vocabulary V equals 7, each row an ID number 1 to 7 beside a token box — h, u, g, p, s border-outlined, ug and hug and tagged from merge 1 and from merge 2. Right: the pipeline on the word pugs, three stages top to bottom joined by arrows — the word pugs, then boxes p ug s with ug highlighted, then three chips 4 6 5, annotated this list of integers is what the model reads, and a note a real vocabulary is about 50,000 rows.
Vocabulary IDs turn pugs into [4, 6, 5].

Our V=7 menu assigns h=1, u=2, g=3, p=4, s=5, ug=6, hug=7. Therefore hugs becomes [7,5], and pugs becomes [4,6,5]. To decode, read the same table backward.

A larger vocabulary might have 50,257 entries or hundreds of thousands. It changes the menu's size, not what an ID means. The learned vector associated with an ID is a separate object.

See it happen

Watch the two merges, then pause before pugs becomes IDs. The first merged piece, ug, has ID 6.

Watch (1:33): what to notice. u+g wins 20-to-15, then hug forms. At the end the never-seen word pugs is chopped by replaying the merges into [4, 6, 5], the numbers from this chapter.

Quirks you can feel in the chatbot

Two panels split by a vertical hairline. Left, headed what you see: the word strawberry in large text with its three r letters highlighted, annotated ten letters, three r's. An arrow labeled tokenizer crosses the divider left to right, with a dashed line and an x glyph annotated letters don't make the trip. Right, headed what the model sees: two ID chips, 3504 tagged straw and 8154 tagged berry, annotated two catalog numbers, letters not included.
Token IDs do not give one slot per letter.

Tokenization can make character tasks less direct, but it does not make them impossible. Models can learn spelling relationships, and different tokenizers split “strawberry” differently. The figure's [straw][berry] split and IDs are teaching examples.

Text changes that can change tokenization
CaseWhat to expect
Leading spaces“ the” and “the” may use different tokens and IDs.
Numbers12345 might split as [123][45], with boundaries unrelated to place value.
LanguagesThe same message can need 20 tokens in one language and 40 in another, depending on the learned menu.
Capacity and usageContext and generation are counted in tokens. At the rough English ratio, 200,000 tokens correspond to about 150,000 words.

These are tendencies to check with the particular tokenizer, not universal splits. Shared pieces let the system handle novel strings, while unequal segmentation makes some texts more expensive to process.

Why the LLM cares

$$ p(\text{next} \mid \text{context}) $$

In words: the probability of each possible next token, given the tokens so far.

For the text path we are studying, token IDs enter and leave the model. The output distribution has one share per vocabulary entry: V shares. An ID selects a piece; it does not encode similarity to another piece.

We need one more conversion before the vector tools can help: replace each ID with a learned arrow.

What you now know

  • Tokens are reusable text units.
  • BPE repeatedly merges frequent adjacent pieces.
  • Our merges produce hug and ug, allowing unseen pugs to be segmented.
  • A token ID identifies a vocabulary entry without measuring meaning.
  • Spaces, spelling, numbers, and language affect token counts.

Where we're headed

We have catalog numbers. Next we'll give each token a vector so geometry can express relationships between pieces of language.