Part III · Ch. 16 — The Recurrent Network (RNN, GRU)

Part III · Chapter 16 of 29

The Recurrent Network (RNN, GRU)

Carry a hidden state through a sequence


What shape does it eat?

A list of measurements arrives in time order. We want to predict the next one, and yesterday’s position in the list matters as much as yesterday’s value. How do you reuse a model without pretending each position stands alone? Carry a small numeric state through the list and watch what happens to it.

A sequence is an ordered list of inputs. A time step is one position in that list, which is not the same as an optimizer step that changes weights. A recurrent network repeats a shared update using the previous hidden state. The hidden state is the vector carried from one position to the next. The stream GRU in this chapter uses a window of the twenty preceding values of a numerical series to predict the next value.

A numerical window of past values with its next-value target, beside a synthetic name split into character tokens. Time runs left to right, and a split boundary sits between windows so none crosses it.
Sequence order is part of the input.

The input axes are sample, sequence position, and feature. For this numerical task the feature axis has length one. Chronological order is preserved, and windows are built separately on each side of the raw-data split. A window crossing the split would quietly carry evaluation history into a training example. Shuffling independent training windows later is a different operation from shuffling the time positions inside a window.

A repeated rule carries a state

Unrolling means drawing repeated state updates as an ordered chain. Several copies of the calculation appear, and there is one shared set of parameters behind them. Each copy’s state output becomes the next copy’s input. That creates a dependency a recurrent forward pass has to respect: no later state finishes before the one in front of it.

Three copies of the hand update on inputs 1, 0 and −1, linked by their states; the matrix symbols in each copy connect to one shared parameter table. A separate panel shows a gate mixing old and candidate state half and half.
Unrolling repeats the calculation, not the parameter set.

A gate is a learned mixing fraction that controls how much state is kept and how much is replaced. A gated recurrent unit uses reset and update gates. The reset gate controls the old-state contribution used to build a candidate; the update gate controls how much old state to keep. The stream GRU stores its parameter rows in reset, update, candidate order, with separate input and hidden matrices and biases.

Three positions, the same weights

Take a two-unit hand recurrence, simpler than the trained gated model. Start with hidden state (0,0). The scalar inputs are 1,0,−1. The input weights are (1,−1), the recurrent matrix is the identity, and both biases are zero. So each update applies tanh to (x,−x) plus the old state. Every input and weight is editable in the hand widget.

Three updates with one parameter set

Let t=tanh(1) and u=tanh(t). The first state is (t,−t), because the old state contributes zero. With the next input zero, the second state is (u,−u). With the final input −1, the third state is (tanh(u−1),tanh(1−u)). Evaluating tanh(z)=(exp(2z)−1)/(exp(2z)+1) gives (0.7616,−0.7616), then (0.6420,−0.6420), then (−0.3434,0.3434).

The signs flip on the last input, because the new negative contribution outweighs the retained positive coordinate. The weights never changed across those three positions. Let bold x at position t be the input vector, bold h the hidden state, and bold W with subscripts x and h the input and recurrent weights. Bold b is the bias. In the separate gate formula, g is the fraction of old state retained:

$$\mathbf{h}_t=\tanh(\mathbf{W}_x \mathbf{x}_t+\mathbf{W}_h\mathbf{h}_{t-1}+\mathbf{b})$$

In words: the vector x at position t holds the current input; mix it with the previous state, add the bias, and bend the result. The hand example above has only one input feature.

$$h_{\mathrm{new}}=g h_{\mathrm{old}}+(1-g)h_{\mathrm{candidate}}$$

In words: keep the gate’s share of the old state and the remaining share of the candidate.

A gate and a shrinking derivative

With old state 1, candidate −1, and keep-gate 1/2, the next state is (1/2)×1+(1−1/2)×(−1)=0. Keeping more of the old state pulls the result toward 1; keeping less pulls it toward −1. Separately, a local slope of 1/2 repeated through three links returns a derivative of (1/2)³=1/8. The first calculation is a state mixture, the second a gradient product.

Compare state and prediction while training

The stream GRU has an eight-coordinate state and a final linear readout of its last state. The token model adds an embedding and applies its output head at every position. They are different tasks sharing one recurrent family. So the numerical run is compared with its numerical baseline, and the name run with other models on the same name corpus and split.

Two hidden-state coordinates of the stream GRU across the positions of one window, at the start and at the end of training, with its predictions against the truth in a second panel. Axes are fixed and each frame is labeled with its training step.
State changes and prediction changes can be inspected together.

At optimizer step 2000, the stream GRU’s training loss is 0.00288 and its validation loss 0.00287. The name GRU finishes at a validation loss of 1.33. The numerical loss and the character cross-entropy use different units, so their sizes are not a contest between tasks.

A recurrent model carries a state through a shared update, and gates alter what it retains.

Where this shows up when you train

Ordered streams gain from a shared state update when the recent past helps the next prediction. The question is what history is available at the moment you have to answer. A window built from a future observation makes the score look better and changes the real task. Keep the temporal boundary intact before you ask whether a larger recurrent state would help.

Residuals of the trained stream GRU, prediction minus truth, plotted against sample order, with the largest one highlighted and its prediction, truth and difference labeled.
Even the final model has measurable residual errors.

A residual is prediction minus truth. The figure marks the largest absolute residual among the final saved numerical predictions, taking the first index in a tie, and shows the prediction, the truth, and their difference there. Select another sample in the replay and you can run the same arithmetic elsewhere.

What you now know

  • A recurrent model applies the same state-update rule along a sequence.
  • A gate chooses shares of old and candidate state.
  • The saved predictions show where the trained model still misses.

Where we’re headed

Attention lets each position build its own mixture of other positions instead of leaning on a carried state. Continue the story.