Keep every parameter in view
What happens between an input and a model’s prediction? One small model, held fixed, with every number on the page. No new weights hide behind a diagram: the same seventeen parameters come back when we grade the prediction and work out how to change them.
A class is a named output category. Binary classification chooses between two of them, which we call class 0 and class 1. Our model takes two input numbers, produces three internal numbers, then produces two class scores. A forward pass computes those outputs from the current parameters and leaves the parameters alone.
The complete initial table: W₁=[[1,0],[0,1],[1,−1]], b₁=[0,1,−1], W₂=[[1,−1,0],[0,1,1]], and b₂=[0,0]. Subscript one names the hidden layer and subscript two the output layer. Twelve multiplying weights, five biases. The hidden activation is tanh and the last layer is linear.
The recipe matters as much as the table. A different activation would run a different calculation on the same saved values. The order of inputs and classes matters too. Input position one matches the first column of W₁, and output position zero names class 0. Those conventions keep each number attached to its role.
Begin with x=(1,0)
The hidden preactivation is the sum before the bend. Its first entry is 1×1+0×0+0=1. Its second is 0×1+1×0+1=1. Its third is 1×1−1×0−1=0. The preactivation vector is (1,1,0).
The zero in the input makes several contributions vanish, and the full row recipe still ran. A different input would wake those contributions up without changing the matrix. A zero input and a zero stored weight are different reasons for a product to be zero, and keeping both values around matters once we compute derivatives.
A score is not yet a probability
A probability is a numerical share between zero and one for an outcome. A probability vector lists mutually exclusive outcomes whose shares sum to one. Softmax turns logits into those shares: exponentiate each score, then divide by the sum of all the exponentials. Every class takes part in that shared denominator.
The exponential keeps every share positive for finite logits. A larger score gives a larger exponential and a larger share. Dividing by the common total makes the shares sum to one. The order of operations counts: exponentiating and then dividing is different from dividing the raw scores by their sum, which breaks for many score vectors.
A probability share is the model’s assignment under this calculation. It is not by itself a claim that the model’s confidence is well calibrated on future data. Comparing predictions with observed outcomes comes later. For now we only need the shares from a fixed set of logits.
Softmax for the fixed model
The logits are (0,t). We have exp(0)=1 and exp(t)=2.141688. Their denominator is 3.141688. Dividing gives the probability vector (0.318300,0.681700), rounded to six decimals. Argmax means the index of the largest entry, so the decision rule picks class 1.
For class index i, divide its exponential by the sum over all class indices j. The same denominator serves every class. Change one logit and the denominator changes too, so more than one probability share moves. That coupling turns up again in the loss derivative.
$$ \mathrm{softmax}(\mathbf{z})_i=\frac{e^{z_i}}{\sum_j e^{z_j}} $$In words: exponentiate each score, then divide by the sum so that the shares add to one.
For a different pair of logits, (2,1), softmax gives (e/(e+1),1/(e+1))=(0.731059,0.268941). Subtract 1 from both scores and the shares stay the same, because the common exponential factor cancels between each numerator and the denominator.
That cancellation is a useful numerical trick: subtract the largest score before exponentiating. The largest shifted score is then zero, which keeps the exponentials from growing needlessly. The probabilities are unchanged.
Pause after every operation
Before you advance the interactive, predict the values that should come next. Start with the hidden products, then their sums and biases, then tanh. Every stage uses values already computed. It is the same dependency picture from the parallel-work chapter, attached now to a complete numerical example.
See it move
The stepper opens one operation at a time and keeps the finished intermediate values in view. At the end, compare its six-decimal probabilities with (0.318300,0.681700). Edit an input or a parameter and follow the recalculated pass. Those edits make a hand example of your own; the starting table above stays as it is.
Changing the second input is a good probe, because it starts at zero. It brings the zero input-weight products into view. Changing a bias instead adds an offset whatever the input is. Both can move the final probabilities, and they travel different paths through the same recipe.
The video from the layer chapter shows the same wiring-to-matrix idea. The worked numbers above carry the whole pass: input, preactivation, hidden values, logits, probabilities. If your result differs, compare those stages and find the first disagreement instead of checking only the final class.
Where this shows up when you train
A prediction is the model output a task uses. A decision rule such as argmax turns probability shares into a chosen label. Keeping the two apart pays off when an application wants the probabilities themselves, a different decision threshold, or an account of why a marginal case got the label it did.
Keep the table and the input in mind: x=(1,0), the same initial weights, target class 1. Holding the example fixed is what makes the old and new losses comparable. Change the model and the input at the same time and you can’t say which one moved the number.
What you now know
- A forward pass changes activations, not parameters.
- The fixed model produces logits (0,t) for this input.
- Softmax produces probability shares and argmax chooses a label.
Where we’re headed
Next: supply the correct label and work out how costly this prediction is. Continue to the next chapter.