Skip to the page
HOW MACHINES LEARN
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9

Page 8 of 9

The same block inside a language model

A language model turns each piece of text into a list of numbers and passes it through this block again and again. The block is the board from page 5 with a much wider middle row.

A token becomes a list

A language model reads text as tokens, which are words or pieces of words. Each token is looked up in a stored table and becomes a list of numbers. In the 2017 design that current language models descend from, the list was 512 numbers long. Those lists are learned, by the rule from the last page but one.

What one block does

The model is a stack of identical blocks, and each block has two halves. In the first half, called attention, a token takes in information from the other tokens in the text. In the second half each token is worked on alone, by a network with one middle row. The paper that introduced the design describes that second half in one sentence.

Press Play, then touch the middle slots. The token is widened to four times as many slots, the gate shuts the ones whose totals are negative, and what the open ones add is narrowed back and added to the token. Switch between the two tokens to see different slots open.

Each middle slot is a small rule

The weights into a middle slot are a pattern to look for. The slot's total is large when the token matches the pattern, and the gate keeps the slot shut when it does not. The weights out of the slot are what to add to the token when the slot is open. So each middle slot behaves like one rule of the form: if the token shows this, add that.

Why it widens and then narrows

The two ends are fixed

What the block computes is added back onto the token. To be added to a list of 512 numbers it has to be a list of 512 numbers. The input is the token, so that end is 512 wide too. One analysis of these models describes every part of them as “reading and writing to” a single running list per token.

The middle is the row of rules

The number of middle slots is the number of rules the block can hold, and the number of hinges it has to build shapes with. Every theorem on page 4 is about a network with one middle row, and each one says the fit improves as that row gets wider.

A middle row wider than the input is as old as the hardware. Rosenblatt's 1958 machine went from 400 inputs to 512 middle slots to 8 outputs.

A wide middle protects the token from the gate

A gate zeroes every slot whose total is negative. If the middle row were no wider than the token, much of the token would be lost there. The designers of a 2018 image network gave this as their reason for the same widen, gate, narrow shape:

The factor of four was measured

The 2017 paper gives its sizes with no derivation: 512 at the ends and 2,048 in the middle. Its table of experiments reports the alternatives it tried, scored on a translation task.

Middle rowTranslation scoreWeights in the model
1,02425.453 million
2,04825.865 million
4,09626.290 million

A wider middle scored higher and cost more. Four times the token's width was a workable compromise. Found by trying

From the last block to the next word

After the last block, one more table gives a score to every token in the vocabulary, and the squash from page 6 turns the scores into probabilities. The model picks a token, appends it to the text, and runs the whole stack again to get the one after.

Nothing travels back up the stack while a token is being processed. The loops that do exist are the two already covered. During training, blame goes back up, as on page 5. During writing, the model's own output is fed back in at the top.

CARRY THIS FORWARD

The part that learns is the same at every scale: a table, a gate, a table. A language model makes the middle row thousands of slots wide and stacks the block many times.