Remembering Machines

When order is the whole point

Say “the dog chased the cat,” then “the cat chased the dog.” Same words, opposite meaning. Last chapter’s convolutional neural network (CNN) preserved an image’s spatial neighborhoods; a sequence also needs its order preserved. How can a network read a sentence of unknown length and remember what came before?

A fixed stack could reserve slots for a maximum sentence length, but then we must decide what to do with longer inputs and unused slots in shorter ones. Recurrence gives a different construction: the number of times we reuse the cell follows the sequence length. We do not need a separately learned cell for every possible position.

The fix is to read one element at a time and carry a running summary. That summary is a hidden state. A recurrent neural network, or RNN, updates this vector at each step using the same rule. The state changes as the sentence arrives; the learned rule is shared across its positions.

RNNs powered translation, speech recognition, and phone autocomplete before attention became the main design for large language models. This chapter builds a recurrent cell, follows the path its training signal takes, then shows what a direct connection buys. Recurrence is still useful when measurements arrive one at a time and a small running state is all the job needs.

The loop: one step, repeated

At time step $t$, the new input is $x_t$, the previous memory is $h_{t-1}$, and the updated memory is $h_t$. We use a scalar, one number, to make the arithmetic visible. The input weight $w_x$ and memory weight $w_h$ control their contributions; $b$ is a bias, and $f$ is the activation that squashes their sum.

1. Weight. Multiply the input by $w_x$ and old memory by $w_h$.
2. Add. Combine those contributions and the bias $b$.
3. Activate. Apply $f$; carry its output as the new memory.

In words: blend the weighted input and old memory, add a bias, and squash the result into the memory that feeds the next step.

The same $w_x$, $w_h$, and $b$ are used at every step: one neuron’s rule, repeated. This is weight sharing across time rather than space; reading three words or three hundred changes the number of repetitions, not the cell.

Worked example: remember across a zero

Choose $w_x = 1$, $w_h = 1$, $b = 0$, and the identity activation $f(z) = z$, which is no squash at all, so we can hand-check a running total: $h_t = h_{t-1} + x_t$. Start at $h_0 = 0$ and feed $x = (1, 0, 1)$.

$$h_0 = 0,\quad h_1 = h_0 + x_1 = 1,\quad h_2 = h_1 + x_2 = 1,\quad h_3 = h_2 + x_3 = 2$$

In words: start at zero, add one, add zero, then add one; the final memory is two, the count of ones.

The identity activation makes the example checkable, not representative of every RNN. Real cells usually use $\tanh$, a sigmoid cousin, and carry a vector rather than a single count. Their state can track a subject, an open quotation, or a verb awaiting its object.

The fading memory

“The keys that the man near the old wooden desks by the window … were on the table.” The verb agrees with “keys,” and not with “man,” “desks,” or “window.” The hidden state has to preserve “the subject is plural” across all those intervening words.

Learning that link requires backpropagation to carry blame from the verb to the subject. The chain rule multiplies local slopes along the path. In an unrolled RNN, the path crosses time steps; each factor includes the recurrent weight $w_h$ and the activation’s slope. Use a constant factor of $0.5$ to isolate the effect of repeated attenuation.

This is the sigmoid’s vanishing-gradient problem along time steps instead of layers. Weak backward credit makes distant dependencies hard to learn. It does not mean every stored memory halves at each forward step: real factors vary, though a long product can still shrink or explode.

The long path matters even if the final prediction is only one word. Credit must reach the earlier information that would have helped, or training cannot reliably learn to preserve it.

Gates: a protected memory line

An LSTM, short for “long short-term memory,” adds a second track: a cell state. Picture a conveyor belt carrying information through the sequence. The cell can leave selected information on that belt nearly unchanged, instead of rebuilding every memory through the same squashing operation at every step.

A gate is a learned valve controlled by the sigmoid: it outputs a number between $0$ (block) and $1$ (pass) for each memory component and multiplies that component by it. These are adjustable fractions, not switches an engineer opens by hand. The new input and previous hidden state help determine them.

These names describe different decisions. Forgetting an old fact is not the same as refusing to add a new one, and keeping a fact does not mean exposing all of it at every step. That is why the diagram has a cell-state track and a hidden-state output. We can protect a subject’s number while revealing other information needed for the current word.

This is what backpropagation gains. The memory update combines a retained old state with an addition, and adding new information need not weaken the gradient along the old-state path. With a forget gate near $1$, that path can carry credit far backward. The belt is protected rather than lossless: a forget gate near $0$ does discard information.

This design let recurrent networks learn dependencies hundreds of steps long and supported translation, speech recognition, and text prediction. A gated recurrent unit, or GRU, is a related design with two gates instead of three. The details differ, but the aim is the same: learn when to retain information and when to replace it.

Why attention won

Even an LSTM processes a sequence one step at a time. That leaves two limits, both visible in the unrolled chain.

Let the verb look directly at the subject instead of relying on a summary relayed through all the words between them. That shortens the information path itself, rather than improving how well each link remembers.

Attention weights other positions by relevance and pulls information directly from them. The plural “keys” is one hop from “were,” not ten. Positions can compute these connections in parallel during training. The sister site builds the full mechanism, with queries, keys, values and their weighted combination, in the attention chapter.

Follow the bold connection in the picture. Intervening words no longer have to preserve the subject’s information before the verb can use it, which shortens the route for both information and training credit. The model still has to learn which word matters. Direct access creates the opportunity and nothing more.

“Every position at once” needs one qualification. A next-word model must not peek at future answers during training, so it blocks attention to later positions. Earlier positions are still directly reachable. When generating, it produces one new token at a time; parallel training does not mean it can produce an unknown sentence all at once.

State-space models such as Mamba also explore ways to carry information through sequences efficiently, so recurrence is far from obsolete. The tradeoff is the thing to remember: attention buys direct reach and parallel work across known positions, while a recurrent model buys a compact state and a natural way to process a stream.

Where you'll meet this