Remembering Machines

When order is the whole point

Last chapter's CNN treated an image as a grid and shared one small filter across every position in space. That works because a picture is, at heart, a bag of neighborhoods. But now say a sentence out loud — "the dog chased the cat" — and shuffle it: "the cat chased the dog." Same five words, opposite meaning. For pictures, a bag of pixels is fine. For language, order is the signal, and a network that ignores order is useless.

Two things about a sequence break the plain stack of neurons from the neuron. The first is length: a sentence can be three words or three hundred, but a fixed grid of neurons expects a fixed-size input, and there is no natural number of slots to pour a sentence into. The second is order and context: the meaning of a word depends on what came before it — "bank" after "river" is a muddy edge, "bank" after "money" is a vault — so the network needs some memory of the sequence so far. A grid has neither. It sees everything at once and remembers nothing.

The fix is the spine of this whole chapter. Instead of swallowing the sequence in one gulp, read it one element at a time — the way you are reading this line, left to right — and carry a running summary of everything seen so far. That running summary is a hidden state, and a network built to loop over a sequence this way, updating that memory at each step, is a recurrent neural network, or RNN. One picture holds the whole idea.

Now the honest arc, so this chapter has stakes. RNNs were how machines read for years — translation, speech recognition, the autocomplete on your phone — and then, around 2017, they were almost entirely replaced by a different idea called attention. This chapter builds the RNN for real, shows you the exact weakness that doomed it, and tells the honest, mechanical story of why its successor won. The successor is built in full on the sister site; here we earn the reason it had to exist.

The loop: one step, repeated

Let us build the recurrent cell, and it turns out we already own every piece. At each time step $t$, the cell takes two things — the current input $x_t$ (the new word) and the previous hidden state $h_{t-1}$ (the memory so far) — mixes them with weights, and produces the new hidden state $h_t$. That is a neuron from the neuron, whose two inputs happen to be "the new thing" and "what I remembered," and whose output becomes next step's memory. Written out:

In words: blend a bit of the new input with a bit of the old memory, add a bias, squash the result with an activation $f$, and that squashed number is the updated memory — which then feeds the next step.

Here is the fact that makes it work: the same weights $w_x$, $w_h$, and $b$ are used at every single step. The network does not learn one rule for word 1 and a different rule for word 50; there is only ever one cell, run over and over. This is precisely the weight sharing from seeing machines, moved off the space axis and onto the time axis: one small rule, applied repeatedly. And that is exactly what lets an RNN swallow a sequence of any length — three words or three hundred, it is the same single cell run more or fewer times. The two views in the figure above — the compact "rolled" loop and the "unrolled" chain — are the same machine; we will reason with the unrolled one, where the hidden state passes down the line like a baton.

Now the smallest honest example, with numbers you can check on your fingers. Set the weights so plain you can watch the memory form: $w_x = 1$, $w_h = 1$, $b = 0$, and use no squash at all (the identity, $f(z) = z$). The update collapses to a running total, $h_t = h_{t-1} + x_t$. Start the memory at $h_0 = 0$ and feed the sequence $x = (1, 0, 1)$:

In words: the memory starts at zero; the first input $1$ pushes it to $1$; the middle input $0$ leaves it untouched at $1$; the last input $1$ lifts it to $2$. The final hidden state, $2$, is the count of $1$s in the sequence — the network remembered across the zero in the middle.

Read what just happened, because it is the whole point. Nobody stored the individual inputs; the hidden state carried a summary — here, a running count — and that one number let a later step know something about earlier ones. The identity squash was a convenience for hand-checking; real cells use a genuine activation (usually $\tanh$, a cousin of the sigmoid) and a hidden state that is a whole vector of numbers, not a single one — so no reader should walk away thinking an RNN is literally an adding machine. Swap in that real squash and those extra hidden numbers and the same loop can track far subtler things than a count: whether we are inside a quotation, what the subject of the sentence was, whether a verb is still waiting for its object. Memory is just a vector, carried forward and updated.

The fading memory

Here is a sentence the running-sum toy could never fake. "The keys that the man near the old wooden desks by the window ... were on the table." Is it "was" or "were"? It is "were" — the verb must agree with "keys," ten words back. To get this right, an RNN has to carry the single fact "the subject is plural" in its hidden state across every intervening word, without letting it get overwritten by "man," "desks," "window," and the rest. In practice, plain RNNs lose it. Let us see exactly why, because the reason is the heart of the chapter.

To learn a long-range dependency, training has to send blame backward from the verb all the way to the far-back subject — the same backpropagation from teaching the network, where the chain rule multiplies a local slope at every step along the path from a weight to the loss. In an unrolled RNN, that path runs through every time step between the verb and the subject, and — because it is the same cell each time — each step multiplies the traveling signal by roughly the same factor, the recurrent weight $w_h$. (Real cells also multiply by the activation's slope at each step; one honest factor is enough for the picture.) Multiply the same number by itself many times over and exactly one of two things happens.

Make it concrete. Suppose that per-step factor is about $0.5$. Then the signal that survives back to a distant word is:

In words: five steps back, the signal is already down to three percent of its strength; ten steps back, a thousandth; twenty steps back, about a millionth — effectively zero. The blame from a distant word arrives so faint it cannot teach the network anything about that word.

This is the same vanishing-gradient problem you first met at the sigmoid's flat tails in drawing boundaries — but with a new twist. There, the signal shrank as it passed back through layers; here it shrinks as it passes back through time steps, one multiply per step. And if the factor is a touch bigger than $1$ instead — say $1.5$ — the opposite disaster strikes: $1.5^{20} \approx 3300$, the signal explodes and training blows up. That is the exploding-gradient problem, usually patched by the blunt trick of capping the signal's size when it grows too large. Below $1$ it vanishes; above $1$ it explodes; the knife edge of exactly $1$ is not something training can balance on.

State the consequence plainly, without drama. A plain RNN can reliably learn dependencies a handful of steps apart, but the memory of anything far back fades exponentially — it forgets the beginning of a long sentence by the time it reaches the end. This is not a bug you can tune away by making the network bigger or training it longer; it is baked into the repeated-multiplication structure of the loop itself. The entire next idea exists to defeat it.

Gates: a conveyor belt for memory

The vanishing gradient comes from forcing every memory through the same squashing multiply at every step. So the fix, kept brief because its algebra is the sister site's business, is to stop doing that. A LSTM (short for "long short-term memory") adds a second track: a cell state that runs along the whole sequence like a conveyor belt, edited only by small, gentle, controlled tweaks instead of being rebuilt from scratch each step. Because information can ride that belt almost untouched, its gradient does not get multiplied away — the memory survives.

The valves have names, and we can meet them without drowning in equations. A gate is a valve run by a little sigmoid — the squash-to-between-$0$-and-$1$ function from drawing boundaries — that outputs a number from $0$ (shut) to $1$ (open) for each piece of memory, and multiplies the signal by it. An LSTM has three. The forget gate decides what to wipe off the belt; the input gate decides what new information to add to it; and the output gate decides what of the belt to reveal as this step's hidden state. That is the whole cast — forget, add, reveal.

And here is the one honest sentence about why it beats the plain loop, tying straight back to teaching the network. On the conveyor belt, moving from one step to the next is mostly addition — add a little, erase a little — rather than repeated multiplication by a weight. And addition does not shrink a gradient the way multiplication does: the training signal can travel far back along the belt without vanishing. That single architectural change let networks learn dependencies hundreds of steps long, and LSTMs went on to power a decade of translation, speech recognition, and text prediction. (A close cousin, the GRU, does the same job with two gates instead of three.)

Why attention took the crown

Even a well-behaved LSTM has two limits it cannot shake, and both spring from the same root: it processes a sequence one step at a time, in order. Name the first limit — speed. Because step 50 needs the output of step 49, which needs step 48, the computation is inherently sequential: you cannot work on all the words at once. Modern hardware — the GPUs that train these models — is built to do thousands of things in parallel, and a machine that insists on going word by word leaves most of that power idle. Train on billions of words this way and it becomes painfully slow.

Name the second limit — distance. To connect two far-apart words, an RNN's information has to travel through every step between them, hop by hop, and — even with an LSTM's gates — something degrades over a long chain. Two words ten apart are ten hops of memory apart. Which raises the question that changed the field: what if any word could look directly at any other word in a single step, no matter how far?

That single step is exactly what attention does, and it is the idea that dethroned recurrence. Instead of passing a memory baton down a chain, attention lets every word compute, in parallel, how much it should attend to every other word, and pull information straight from the relevant ones — the plural "keys" that a verb ten words later needs is now one hop away, not ten. No recurrence, no fading baton, and every position computed at once. The full mechanism — queries, keys, values, and the weighting that combines them — is built step by step on the sister site's chapter on attention; here we need only the reason it won.

End honest, because "RNNs are obsolete" is too glib. Recurrence is not dead: it still fits streaming and low-resource settings, where you truly do process one element at a time as it arrives, and a newer family of recurrence-flavoured models — state-space models like Mamba — is a live research direction precisely because processing all positions at once, as attention does, gets expensive when sequences run to hundreds of thousands of tokens. The pendulum may yet swing back. But for the language models of the last several years, attention won, decisively, for the two mechanical reasons above.

Where you'll meet this

Step back and place the chapter in the story. You now understand the two dominant ways to handle a sequence: recurrence (read it in order, carry a memory) and attention (let every position see every other at once). Everything that reads text, audio, or any ordered stream uses one of these — and knowing why attention displaced recurrence is knowing the single most important architectural shift in modern AI.

Draw the sister-site thread explicitly. The large language models on the sister site are, at heart, giant attention machines — the transformer (see the sister site's attention chapter and the ones around it) is what you get when you take the "every word looks at every other word" idea, stack it dozens of layers deep, and train it on much of the internet. The reader who understands why an RNN's memory fades understands exactly what problem the transformer was built to solve.

Close warm. You have run a recurrent cell by hand — a running count, $0 \to 1 \to 1 \to 2$, remembering across a zero — watched a memory fade exponentially to a millionth of its strength across twenty steps, seen how a gated conveyor belt rescues it, and learned the honest, mechanical reason attention won: parallelism and reach, not magic. That is the whole arc of how machines learned to read.