Page 8 of 9
The same block inside a language model
A language model turns each piece of text into a list of numbers and passes it through this block again and again. The block is the board from page 5 with a much wider middle row.
A token becomes a list
A language model reads text as tokens, which are words or pieces of words. Each token is looked up in a stored table and becomes a list of numbers. In the 2017 design that current language models descend from, the list was 512 numbers long. Those lists are learned, by the rule from the last page but one.
What one block does
The model is a stack of identical blocks, and each block has two halves. In the first half, called attention, a token takes in information from the other tokens in the text. In the second half each token is worked on alone, by a network with one middle row. The paper that introduced the design describes that second half in one sentence.
Each middle slot is a small rule
The weights into a middle slot are a pattern to look for. The slot's total is large when the token matches the pattern, and the gate keeps the slot shut when it does not. The weights out of the slot are what to add to the token when the slot is open. So each middle slot behaves like one rule of the form: if the token shows this, add that.
Why it widens and then narrows
The two ends are fixed
What the block computes is added back onto the token. To be added to a list of 512 numbers it has to be a list of 512 numbers. The input is the token, so that end is 512 wide too. One analysis of these models describes every part of them as “reading and writing to” a single running list per token.
The middle is the row of rules
The number of middle slots is the number of rules the block can hold, and the number of hinges it has to build shapes with. Every theorem on page 4 is about a network with one middle row, and each one says the fit improves as that row gets wider.
A middle row wider than the input is as old as the hardware. Rosenblatt's 1958 machine went from 400 inputs to 512 middle slots to 8 outputs.
A wide middle protects the token from the gate
A gate zeroes every slot whose total is negative. If the middle row were no wider than the token, much of the token would be lost there. The designers of a 2018 image network gave this as their reason for the same widen, gate, narrow shape:
The factor of four was measured
The 2017 paper gives its sizes with no derivation: 512 at the ends and 2,048 in the middle. Its table of experiments reports the alternatives it tried, scored on a translation task.
| Middle row | Translation score | Weights in the model |
|---|---|---|
| 1,024 | 25.4 | 53 million |
| 2,048 | 25.8 | 65 million |
| 4,096 | 26.2 | 90 million |
A wider middle scored higher and cost more. Four times the token's width was a workable compromise. Found by trying
From the last block to the next word
After the last block, one more table gives a score to every token in the vocabulary, and the squash from page 6 turns the scores into probabilities. The model picks a token, appends it to the text, and runs the whole stack again to get the one after.
Nothing travels back up the stack while a token is being processed. The loops that do exist are the two already covered. During training, blame goes back up, as on page 5. During writing, the model's own output is fed back in at the top.
CARRY THIS FORWARD
The part that learns is the same at every scale: a table, a gate, a table. A language model makes the middle row thousands of slots wide and stacks the block many times.