What shape does it eat?
Two spirals wind around the same center. Can we tell which arm a point belongs to from its two coordinates alone? We know how to multiply a matrix, apply a bend, and update weights. Those operations are enough to build a boundary that follows the data.
A dense layer connects every output to every input in the layer before it. A multilayer perceptron stacks those layers with activations between them. “Dense” describes the connections, and the points themselves can be spread far apart. A decision boundary is the set of input locations where the selected class changes. In our two-class problem it separates the region assigned to class zero from the region assigned to class one.
Each point gives one row with two features. Its label is kept separately, so the model can’t read the answer as an input. The spiral network we trained has widths 2→16→16→2. Its weight arrays have shapes (16,2), (16,16), and (2,16), with a bias for every output at each stage. The hidden arrays give a point more coordinates to work with before it shrinks to two scores.
Move the space before drawing the line
A straight boundary in the original plane can’t follow both interleaved arms. Rather than make the final decision complicated, change the space it sees. Each hidden layer remixes the coordinates, shifts them, and bends the result with tanh. The final layer produces a pair of logits, the scores before softmax, and comparing those two numbers picks a class.
Follow one labeled lattice point between the two panels. Its identity holds; its coordinates change. In the final score plane the decision is the straight line where the two logits are equal, and points above it have the larger class-one score. Pull that comparison back through the network and it can form a curved boundary in the input plane.
The recorded tab replays the spiral network. Select an input point and it computes logits from the latest saved parameter snapshot and shows that snapshot’s step. Boundary pictures were saved more often than full parameter sets, so the boundary shown can be from a later step than the parameters. The editable hand tab starts a separate calculation with its own loss history and starting weights.
A small forward pass we can check
Back to the small network from the forward pass. The input vector is (1,0). The first weight rows are (1,0), (0,1), and (1,−1), with biases 0,1,−1. The two output rows are (1,−1,0) and (0,1,1), with zero output biases. Seventeen parameters, small enough to hold in view at once.
Every intermediate value
The first sums are 1×1+0×0+0=1, then 0×1+1×0+1=1, and finally 1×1+(−1)×0−1=0. Let t=(e²−1)/(e²+1), the exact expression for tanh(1). The hidden vector is (t,t,0). The output rows give t−t+0=0 and 0+t+0=t. So the logits are (0,t), and the probability of class one is exp(t)/(1+exp(t)). That is about 0.682.
For a general hidden layer, let bold h name its vector of current values, bold W its weight matrix, and bold b its bias vector. The subscript k says which layer is being applied. The next layer’s hidden values come from the same recipe:
$$\mathbf{h}_{k+1}=\tanh(\mathbf{W}_k\mathbf{h}_k+\mathbf{b}_k)$$In words: each hidden layer remixes the preceding values, shifts them, then bends them.
Replay the real run
Training now reads as changes in those parameters rather than a colored region drifting around. At initialization the network already has a boundary, because its current weights already produce scores. Learning moves that boundary by evaluating batches, computing derivatives, and updating the same parameter arrays. The split stays fixed through the replay.
At step 2000 the training loss is 7.9e-05, the validation loss is 5.6e-05, and validation accuracy is 1.0: every held-out point on the generated spiral is classified correctly.
Where this shows up when you train
A dense model is a good starting point for a table of features, because it can combine columns without assuming that neighboring columns are neighboring places. The columns still need a fixed order: swap two features without swapping their weights and the computation changes. The model knows which input slot it receives, not the human meaning of a CSV header.
The hand failure is small enough to see at once. A weight row (1,0) applied to input (1,0) gives 1×1+0×0=1. Move that nonzero entry to the other input slot, making (0,1), and the same row gives 1×0+0×1=0. The pattern moved and the position-specific rule stayed put. A larger dense image classifier can learn several positions, and it still doesn’t get a shared local rule for free.
Keep representation and optimization apart. Width and activation set which transformations the model can express. The data, objective, learning rate, and update budget affect which one it reaches. The spiral run shows one combination working on one task. The other widths and rates in the 84-trial sweep of chapter 21 are separate runs with their own histories.
What you now know
- Dense layers combine all preceding entries.
- The final class boundary compares the output scores.
- A recorded successful split does not establish performance on every input.
Where we’re headed
Convolution shares a local calculation, so moving a pattern doesn’t demand a different set of weights. Continue the story.