This question came from Matthew Reid of Plano, Texas, and I am glad he asked it, because the word vocabulary is doing something sneaky here. The short answer is that the count is a budget somebody chose, a set number of reusable pieces, rather than a dictionary somebody transcribed. So no, it is not one entry per word per language; no language would fit, and the list would never close. And no, training on English alone would not shrink it much: the most famous English-filtered model in the field carries over fifty thousand entries. Let me walk you from a five-letter toy up to a table you can weigh in mebibytes.
An inventory, not a dictionary
A person's vocabulary is the words they know. A model's vocabulary is nothing of the kind. It is a parts inventory: a fixed, numbered list of text pieces, each with an integer address, and that is the entire contents. Text going in is cut into those pieces, and text coming out is assembled from them.
Each entry is a token: a whole word sometimes, a fragment of one often, a space, a punctuation mark, a stretch of code, a single byte. The program that does the cutting is the tokenizer, and the vocabulary is its parts list. The addresses are addresses and nothing more. Entry 500 is no closer in meaning to entry 501 than to entry 9,000.
Types and occurrences: two numbers people merge into one
Most of the confusion in Matthew's question dissolves once two counts are kept apart. A type is a distinct entry in the inventory; an occurrence is one position in a text. The vocabulary counts types. Your prompt spends occurrences.
Take a three-word training text: low low lower. Cut into letters, it has eleven occurrences: eleven positions to fill, reading left to right. It only ever uses five distinct letters: l, o, w, e and r. Five types, eleven occurrences, one text.
The same split holds at full scale: a ten-thousand-token document reuses a few thousand entries over and over. The number of types is written $V$, the vocabulary size. That is the same $V$ that shows up as the number of scores the model produces for each next-token decision.
Why pieces, and not words
So what size should the parts be? Three options, and the middle one wins.
Whole words. Tempting, and hopeless. The list never closes: names, compounds, inflections, product numbers, slang, typos and words invented yesterday all keep arriving. An unabridged English dictionary runs to hundreds of thousands of entries and still misses most proper nouns. Worse, a word list has a cliff at the edge. Anything not on it cannot be represented at all. And plenty of writing systems do not put spaces between words, so "the word" is not even a well-defined unit to list.
Letters, or bytes. A byte is the eight-binary-digit unit every computer stores text in, with 256 possible values. This is the opposite extreme, and it never fails: a couple of hundred entries cover everything anyone can type. The price is length. Every position is a separate step of work, and cutting this finely makes sequences several times longer for no gain in meaning.
Reusable pieces. The middle. Keep the small units so nothing is unrepresentable, then spend extra entries on fragments that recur often enough to earn a slot: ing, tion, ly, common whole words, a space followed by the. Familiar text gets short; unfamiliar text still gets written, one small piece at a time.
That middle row is subword tokenization, and it is what every current large model uses. The full mechanism gets a slower treatment in Text Becomes Numbers; here I only want the count.
A miniature tokenizer, built by hand
Where do the pieces come from? Nobody writes them down. They are learned from text by the field's most common method, byte-pair encoding, or BPE. The whole algorithm fits in one sentence: find the most frequent adjacent pair and glue it together into a new entry, then do it again.
Two merges on "low low lower"
Our whole training text is low low lower. Spaces are fixed word boundaries here, so no merge crosses one, and we start from letters. That is 5 types and 11 occurrences.
Count adjacent pairs inside words: l+o appears three times, o+w three times, w+e once, e+r once. There is a tie at the top, and our rule breaks it toward l+o.
Merge one. Add lo to the inventory. The text becomes [lo][w] [lo][w] [lo][w][e][r]. The inventory has gone from 5 entries to 6; the text has gone from 11 positions to 8.
Merge two. Now lo+w appears three times, so add low. The text becomes [low] [low] [low][e][r]. Inventory 7; text 5.
In words: each merge buys one new entry and saves three positions, because the pair it glued together was sitting in the text three times. The two counters move in opposite directions on every step, and that opposition is the entire economics of a vocabulary. Stop the loop at 7 entries and you have a 7-entry vocabulary; keep going for fifty thousand merges and you have a real one.
One thing to hold on to: this learning happens once, before the model is trained, on a sample of text, and then the merge list is frozen. A production tokenizer does not add an entry when you type a word it has never seen; it spells that word out of the pieces it already has.
The count is a budget, and English does not set it
This part answers Matthew's second and third questions directly. Two decisions are made independently, and people collapse them into one:
- What text the tokenizer is trained on: English only, fifty languages, code and prose, and in what proportions.
- How many entries you budget: a target number typed into a configuration file before the merging loop starts. The loop runs until it hits that number and then stops.
And neither is the text the model itself is later trained on. The tokenizer learns from its own sample, is frozen, and the model then trains against that frozen list. So "train the model on English" does not touch the tokenizer at all.
The second decision is not computed from the first. It is chosen against the costs in the next section, and the same budget can be spent very differently. Four published examples:
GPT-2's corpus was English by construction. Its authors wrote that they "deliberately removed non-English webpages from WebText as a filtering step", and a language detector found only about 10 MB of French left in it. Its vocabulary is 50,257 entries (GPT-2 paper, sections 2.2 and 3.7). Llama 2, which is not English-only, states a total vocabulary size of 32,000 (Llama 2 paper, section 2.2). Llama 3's card says its tokenizer has a vocabulary of 128K tokens (Llama 3 model card), and Qwen2.5 declares 152,064 (its config.json). The English-filtered one is not the small one.
There is one more trap inside "train it on English only." If you take an existing model and train it further on English text, its vocabulary does not shrink. The unused entries sit there. Shrinking it is deliberate surgery on the tables, and not something the data does on its own.
What a big inventory costs, and what it buys
Every entry has to be paid for, and the bill arrives in a table. Each of the $V$ entries owns one row of numbers, its embedding, and each row is $d$ numbers wide. Take a mid-sized model with $V = 32{,}000$ and $d = 1{,}024$:
$$N = V \times d = 32{,}000 \times 1{,}024 = 32{,}768{,}000$$In words: multiply the number of entries by the width of a row and you get the number of stored values in the table: a little under thirty-three million of them, for the input table alone.
Now weigh it. The byte came up when we were sizing the pieces: eight binary digits, two hundred and fifty-six possible values. These models usually keep each stored value in two of them. A mebibyte, written MiB, is 1,048,576 bytes, the binary cousin of the megabyte. Windows reports storage this way; macOS counts a plain million.
$$32{,}768{,}000 \times 2 = 65{,}536{,}000 \text{ bytes} = 62.5 \text{ MiB}$$In words: two bytes for every value gives just over sixty-five million bytes, which divided by 1,048,576 comes to sixty-two and a half mebibytes. That is one table, for one model, before a single layer of the actual network.
And the vocabulary is charged twice: the same rows are read again at the far end. Some models keep a second copy of the table there and some reuse the first, and either way one score per entry must be computed before every next piece. A wider vocabulary means more to sort through for every token written.
So what does the spending buy? Shorter sequences. Every merge that makes text shorter makes each step cheaper, and the cost of attention, the step where every position in the text looks at every other position, grows with the square of the sequence length, because every position looking at every other is length times length. Halve the length and you quarter that bill. Shorter sequences also mean more of your document fits in the context window, the fixed number of positions the model can read at once. The trade has a far end, though: push the budget high enough and the extra entries are rare pieces that each show up in training too seldom to learn a good row, so you are paying full price for rows that never get educated.
Bytes are the floor, so nothing is ever missing
One worry usually survives all of this: what happens when text arrives that the inventory cannot spell? It cannot happen, and the reason is the floor underneath every modern vocabulary.
All digital text is already bytes. A byte takes one of 256 values, and the UTF-8 encoding writes every character of every writing system (Cyrillic, Arabic, Han, emoji, mathematical symbols) as a sequence of bytes. So a vocabulary that includes all 256 byte values can spell anything at all. GPT-2's authors built that, noting that "a byte-level version of BPE only requires a base vocabulary of size 256", and Llama 2's tokenizer likewise falls back to bytes "to decompose unknown UTF-8 characters".
That is why "out of vocabulary" does not happen to these models. A rare string is not rejected. It is spelled the long way, in more pieces. The cost of the unfamiliar is length rather than failure, and length is the same currency the merges were buying back.
See it move
The video walks the same ground more slowly: the counters moving apart merge by merge, one phrase re-cut at three grains, the table filling to 62.5 MiB. Watch the counters rather than the words. They are the argument.
What you now know
- A model's vocabulary is a numbered inventory of reusable text pieces, not a dictionary of words it knows, and a token ID is an address rather than a meaning.
- Types and occurrences are different counts: the inventory holds types and a text spends occurrences, so repeating a piece costs a position and not an entry.
- Whole words cannot be listed because the list never closes, and single letters make sequences needlessly long, so the pieces sit in between, learned by merging the most frequent adjacent pair over and over.
- Every merge buys one entry and shortens the text, which is why tens of thousands of entries is the settled answer to a trade rather than a count of anything.
- The size is a budget typed into a configuration file, decided separately from which languages the tokenizer saw. GPT-2's English-filtered corpus produced 50,257 entries, Llama 2's many-language one 32,000.
- The bill is a table of $V$ rows by $d$ numbers: 32,000 by 1,024 at two bytes a number is 62.5 MiB for the input table alone, plus one output score per entry at every step, bought in exchange for shorter sequences.
- Because all 256 byte values are in the inventory, nothing is ever out of vocabulary. Unfamiliar text costs extra positions rather than a failure.
Where we're headed
So, Matthew: tens of thousands of entries, because that is roughly where the merging stops paying. Nobody counted the words in a language. Two loose ends are worth pulling. The first is those rows. I said each entry owns a row of numbers and moved on, but the rows are where the model's grip on language lives, and how a row gets learned is its own question on this site. The second is what happens at the far end of the pipeline, where the model produces one score for every entry in this inventory and has to turn those scores into a single choice. That is the question we started with. The vocabulary is the list those scores are a list over. Now you know where the list came from.