Part I · Ch. 7 — The Forward Pass, Every Number

Part I · Chapter 7 of 29

The Forward Pass, Every Number

Follow an input through two layers


Keep every parameter in view

What happens between an input and a model’s prediction? One small model, held fixed, with every number on the page. No new weights hide behind a diagram: the same seventeen parameters come back when we grade the prediction and work out how to change them.

A class is a named output category. Binary classification chooses between two of them, which we call class 0 and class 1. Our model takes two input numbers, produces three internal numbers, then produces two class scores. A forward pass computes those outputs from the current parameters and leaves the parameters alone.

W1, b1, W2, and b2 are displayed as arrays of shapes (3,2), (3), (2,3), and (2). Six plus three plus six plus two entries give seventeen parameters.
These seventeen parameters remain the same through the forward and loss examples.

The complete initial table: W₁=[[1,0],[0,1],[1,−1]], b₁=[0,1,−1], W₂=[[1,−1,0],[0,1,1]], and b₂=[0,0]. Subscript one names the hidden layer and subscript two the output layer. Twelve multiplying weights, five biases. The hidden activation is tanh and the last layer is linear.

The recipe matters as much as the table. A different activation would run a different calculation on the same saved values. The order of inputs and classes matters too. Input position one matches the first column of W₁, and output position zero names class 0. Those conventions keep each number attached to its role.

Begin with x=(1,0)

The hidden preactivation is the sum before the bend. Its first entry is 1×1+0×0+0=1. Its second is 0×1+1×0+1=1. Its third is 1×1−1×0−1=0. The preactivation vector is (1,1,0).

The zero in the input makes several contributions vanish, and the full row recipe still ran. A different input would wake those contributions up without changing the matrix. A zero input and a zero stored weight are different reasons for a product to be zero, and keeping both values around matters once we compute derivatives.

Use the same intermediate values

The same activation now goes on each of those three sums. Tanh maps zero to zero, and the two sums equal to one give identical nonzero values. Call that repeated value t, so the next calculation stays readable. The letter abbreviates one fixed number; it is not another parameter.

Input (1,0) becomes pre-activation (1,1,0), hidden values (t,t,0), and logits (0,t), where t = 0.761594. The corresponding shapes are (2), (3), (3), and (2).
The shape tells us how many values each stage must produce.

Let t=tanh(1)=(e²−1)/(e²+1)=0.761594, rounded for display. The hidden vector is (t,t,0). The first two units have the same activation for this input even though their weight rows and biases differ. That equality holds for this one pass; it doesn’t mean the units always behave alike.

The next layer takes the hidden values as its input vector. It uses three entries because W₂ has three columns, and its two rows produce two outputs. Reading shapes along the diagram confirms every stage gets the number of values it expects, before the numerical answer is in question.

Compute the two output scores

W₂h+b₂ gives (1t−1t+0×0,0t+1t+1×0)=(0,t). The first row cancels the two equal hidden values. The second keeps the second hidden value and adds zero. These unsquashed scores are logits. The first is the score for class 0, the second for class 1.

Let h name the hidden vector and z the logits. Each bold symbol refers to a whole list. The first expression runs the hidden affine transformation and tanh. The second runs the final affine transformation. No output activation appears there, because those values are still raw scores.

$$ \mathbf{h}=\tanh(\mathbf{W}_1\mathbf{x}+\mathbf{b}_1),\qquad\mathbf{z}=\mathbf{W}_2\mathbf{h}+\mathbf{b}_2 $$

In words: compute the hidden values with hyperbolic tangent, then mix them into the final scores.

A logit can be negative, zero, or positive. It needn’t lie between zero and one, and two logits needn’t add to one. So a score of zero doesn’t mean the model gives that class zero probability. Getting probability shares takes another transformation, stated explicitly.

A score is not yet a probability

A probability is a numerical share between zero and one for an outcome. A probability vector lists mutually exclusive outcomes whose shares sum to one. Softmax turns logits into those shares: exponentiate each score, then divide by the sum of all the exponentials. Every class takes part in that shared denominator.

Three panels show class scores (0,0.761594), exponentials (1,2.141688), and probability shares (0.318300,0.681700). Each has its own vertical scale; class 1 has the larger final share.
Softmax turns scores into shares; argmax selects the largest share.

The exponential keeps every share positive for finite logits. A larger score gives a larger exponential and a larger share. Dividing by the common total makes the shares sum to one. The order of operations counts: exponentiating and then dividing is different from dividing the raw scores by their sum, which breaks for many score vectors.

A probability share is the model’s assignment under this calculation. It is not by itself a claim that the model’s confidence is well calibrated on future data. Comparing predictions with observed outcomes comes later. For now we only need the shares from a fixed set of logits.

Softmax for the fixed model

The logits are (0,t). We have exp(0)=1 and exp(t)=2.141688. Their denominator is 3.141688. Dividing gives the probability vector (0.318300,0.681700), rounded to six decimals. Argmax means the index of the largest entry, so the decision rule picks class 1.

For class index i, divide its exponential by the sum over all class indices j. The same denominator serves every class. Change one logit and the denominator changes too, so more than one probability share moves. That coupling turns up again in the loss derivative.

$$ \mathrm{softmax}(\mathbf{z})_i=\frac{e^{z_i}}{\sum_j e^{z_j}} $$

In words: exponentiate each score, then divide by the sum so that the shares add to one.

For a different pair of logits, (2,1), softmax gives (e/(e+1),1/(e+1))=(0.731059,0.268941). Subtract 1 from both scores and the shares stay the same, because the common exponential factor cancels between each numerator and the denominator.

That cancellation is a useful numerical trick: subtract the largest score before exponentiating. The largest shifted score is then zero, which keeps the exponentials from growing needlessly. The probabilities are unchanged.

Pause after every operation

Before you advance the interactive, predict the values that should come next. Start with the hidden products, then their sums and biases, then tanh. Every stage uses values already computed. It is the same dependency picture from the parallel-work chapter, attached now to a complete numerical example.

See it move

The stepper opens one operation at a time and keeps the finished intermediate values in view. At the end, compare its six-decimal probabilities with (0.318300,0.681700). Edit an input or a parameter and follow the recalculated pass. Those edits make a hand example of your own; the starting table above stays as it is.

Changing the second input is a good probe, because it starts at zero. It brings the zero input-weight products into view. Changing a bias instead adds an offset whatever the input is. Both can move the final probabilities, and they travel different paths through the same recipe.

A layer’s connections, matrix and computation are the same recipe.

The video from the layer chapter shows the same wiring-to-matrix idea. The worked numbers above carry the whole pass: input, preactivation, hidden values, logits, probabilities. If your result differs, compare those stages and find the first disagreement instead of checking only the final class.

Where this shows up when you train

A prediction is the model output a task uses. A decision rule such as argmax turns probability shares into a chosen label. Keeping the two apart pays off when an application wants the probabilities themselves, a different decision threshold, or an account of why a marginal case got the label it did.

Keep the table and the input in mind: x=(1,0), the same initial weights, target class 1. Holding the example fixed is what makes the old and new losses comparable. Change the model and the input at the same time and you can’t say which one moved the number.

What you now know

  • A forward pass changes activations, not parameters.
  • The fixed model produces logits (0,t) for this input.
  • Softmax produces probability shares and argmax chooses a label.

Where we’re headed

Next: supply the correct label and work out how costly this prediction is. Continue to the next chapter.