What shape does it eat?
A list of measurements arrives in time order. We want to predict the next one, and yesterday’s position in the list matters as much as yesterday’s value. How do you reuse a model without pretending each position stands alone? Carry a small numeric state through the list and watch what happens to it.
A sequence is an ordered list of inputs. A time step is one position in that list, which is not the same as an optimizer step that changes weights. A recurrent network repeats a shared update using the previous hidden state. The hidden state is the vector carried from one position to the next. The stream GRU in this chapter uses a window of the twenty preceding values of a numerical series to predict the next value.
The input axes are sample, sequence position, and feature. For this numerical task the feature axis has length one. Chronological order is preserved, and windows are built separately on each side of the raw-data split. A window crossing the split would quietly carry evaluation history into a training example. Shuffling independent training windows later is a different operation from shuffling the time positions inside a window.
A repeated rule carries a state
Unrolling means drawing repeated state updates as an ordered chain. Several copies of the calculation appear, and there is one shared set of parameters behind them. Each copy’s state output becomes the next copy’s input. That creates a dependency a recurrent forward pass has to respect: no later state finishes before the one in front of it.
A gate is a learned mixing fraction that controls how much state is kept and how much is replaced. A gated recurrent unit uses reset and update gates. The reset gate controls the old-state contribution used to build a candidate; the update gate controls how much old state to keep. The stream GRU stores its parameter rows in reset, update, candidate order, with separate input and hidden matrices and biases.
Three positions, the same weights
Take a two-unit hand recurrence, simpler than the trained gated model. Start with hidden state (0,0). The scalar inputs are 1,0,−1. The input weights are (1,−1), the recurrent matrix is the identity, and both biases are zero. So each update applies tanh to (x,−x) plus the old state. Every input and weight is editable in the hand widget.
Three updates with one parameter set
Let t=tanh(1) and u=tanh(t). The first state is (t,−t), because the old state contributes zero. With the next input zero, the second state is (u,−u). With the final input −1, the third state is (tanh(u−1),tanh(1−u)). Evaluating tanh(z)=(exp(2z)−1)/(exp(2z)+1) gives (0.7616,−0.7616), then (0.6420,−0.6420), then (−0.3434,0.3434).
The signs flip on the last input, because the new negative contribution outweighs the retained positive coordinate. The weights never changed across those three positions. Let bold x at position t be the input vector, bold h the hidden state, and bold W with subscripts x and h the input and recurrent weights. Bold b is the bias. In the separate gate formula, g is the fraction of old state retained:
$$\mathbf{h}_t=\tanh(\mathbf{W}_x \mathbf{x}_t+\mathbf{W}_h\mathbf{h}_{t-1}+\mathbf{b})$$In words: the vector x at position t holds the current input; mix it with the previous state, add the bias, and bend the result. The hand example above has only one input feature.
$$h_{\mathrm{new}}=g h_{\mathrm{old}}+(1-g)h_{\mathrm{candidate}}$$In words: keep the gate’s share of the old state and the remaining share of the candidate.
A gate and a shrinking derivative
With old state 1, candidate −1, and keep-gate 1/2, the next state is (1/2)×1+(1−1/2)×(−1)=0. Keeping more of the old state pulls the result toward 1; keeping less pulls it toward −1. Separately, a local slope of 1/2 repeated through three links returns a derivative of (1/2)³=1/8. The first calculation is a state mixture, the second a gradient product.
Compare state and prediction while training
The stream GRU has an eight-coordinate state and a final linear readout of its last state. The token model adds an embedding and applies its output head at every position. They are different tasks sharing one recurrent family. So the numerical run is compared with its numerical baseline, and the name run with other models on the same name corpus and split.
At optimizer step 2000, the stream GRU’s training loss is 0.00288 and its validation loss 0.00287. The name GRU finishes at a validation loss of 1.33. The numerical loss and the character cross-entropy use different units, so their sizes are not a contest between tasks.
Where this shows up when you train
Ordered streams gain from a shared state update when the recent past helps the next prediction. The question is what history is available at the moment you have to answer. A window built from a future observation makes the score look better and changes the real task. Keep the temporal boundary intact before you ask whether a larger recurrent state would help.
A residual is prediction minus truth. The figure marks the largest absolute residual among the final saved numerical predictions, taking the first index in a tie, and shows the prediction, the truth, and their difference there. Select another sample in the replay and you can run the same arithmetic elsewhere.
What you now know
- A recurrent model applies the same state-update rule along a sequence.
- A gate chooses shares of old and candidate state.
- The saved predictions show where the trained model still misses.
Where we’re headed
Attention lets each position build its own mixture of other positions instead of leaning on a carried state. Continue the story.