The wire is a number
A neural-network drawing has enough connections to look like spaghetti. What gets saved to disk is simpler: every connection gathered into a matrix, one bias per output, and the recipe that turns inputs into outputs. The picture becomes a readable inventory.
Take the matrix [[1,−1,0],[0,1,1]] and bias [0,0]. This is the output layer of the small model we follow through the next several chapters. A weight is a multiplier controlling one input’s contribution. A layer is a set of outputs computed together by one stage. The same input can feed several outputs through different weights.
The matrix row names the destination output and the column names the source input. Follow the connection from input two to output one. It is W₁₂=−1, the second entry of the first row. The wire and the cell are two drawings of one parameter. A zero entry is still part of the table, even though its connection contributes nothing at the current setting.
A positive weight lets a positive input push that row’s sum up. A negative weight pushes it down. That doesn’t make one connection helpful and another harmful; the task decides which combination is useful. Calculate the whole output before reading anything into one wire.
A three-input, two-output layer
Use input (1,1,0). Output one is 1×1−1×1+0×0+0=0. Output two is 0×1+1×1+1×0+0=1. Each row supplies three multiplying weights, and its bias supplies the added offset at the end. The output vector is (0,1).
The same input shows up in both calculations. The input list is not divided between the outputs; each output gets all three entries. That is what dense means here: every destination has a stored multiplying weight for every source input. One of those stored weights may equal zero without changing the layer’s shape.
Warp it, shift it, then bend it
A layer does more than warp and shift a grid. It also applies an activation, a function after the weighted sum that changes how the result responds to its input. To see the effect without adding dimensions, go back to the two-coordinate lattice. Leave its warp as the identity, shift it by (1,−1), then bend each coordinate on its own.
ReLU means rectified linear unit: it returns the larger of zero and its input. The selected point (2,1) becomes (3,0) after the shift and stays (3,0) after ReLU. Points whose shifted horizontal coordinate is negative land on the vertical axis. Points whose shifted vertical coordinate is negative land on the horizontal axis. Different original points can now share one result.
Check an activation value
An exponential extends repeated multiplication to noninteger inputs. The number e, about 2.71828, is the base of the natural exponential. Since e²≈7.389, tanh(1)=(7.389−1)/(7.389+1)≈0.7616. At zero, tanh(0)=0. ReLU(−1)=0, while ReLU(1)=1. The same input gets different answers from different functions.
In our fixed model, the internal layer uses tanh and the output layer is left unsquashed. Let h name the vector of internal outputs, W the layer’s weight matrix, x its input vector, and b its bias vector. Applying tanh to a vector means applying it to each entry on its own, after the matrix multiplication and the addition.
$$ \mathbf{h}=\tanh(\mathbf{W}\mathbf{x}+\mathbf{b}) $$In words: multiply and add, include the bias, then apply hyperbolic tangent to every output.
For the second activation, use a scalar z for one weighted sum before the bend. The expression e raised to −z is an exponential, not multiplication by a variable named e. The sigmoid’s denominator stays positive, and the fraction approaches its two limiting values smoothly.
$$ \sigma(z)=\frac{1}{1+e^{-z}} $$In words: the sigmoid divides one by one plus the exponential of the negative input.
Without a bend between layers, repeated linear maps collapse into one matrix, and repeated affine maps into one matrix plus one bias. Stacking more stages would reorganize the calculation without giving it a new kind of shape. A nonlinear activation breaks that limit. That is what the bend is for.
A model is a list of parameter arrays
A hidden unit is an internal output between the original input and the final prediction. The width of a layer is its number of units. Depth counts successive learned layers. A parameter is a stored value that training may change. Multiplying weights and added biases are both parameters, and leaving the biases out gives the wrong inventory.
Suppose a layer receives n_in input entries and produces n_out output entries. Each output needs one multiplier per input, so the weight table has n_in×n_out entries, plus one bias per output. Those symbols count sizes, not values. Swapping the input (1,1,0) for another three-entry vector leaves the count alone.
Count the complete fixed model
The hidden layer takes two inputs and makes three outputs: 2×3+3=9 parameters. Its weight matrix is [[1,0],[0,1],[1,−1]] and its bias is [0,1,−1]. The output layer takes three values and makes two outputs: 3×2+2=8 parameters. Together, 9+8=17. Twelve are multiplying weights and five are biases.
In words: one multiplying weight per connection, plus one bias per output.
The saved shapes read in order: (3,2), (3), (2,3), (2). The first pair belongs to the hidden layer and the second pair to the output layer. A bias vector’s length matches the number of output rows in its partner matrix. That relationship is a quick consistency check before any arithmetic.
The count covers this model’s stored numbers, not all the memory training needs. Training also keeps intermediate results and derivatives, and an update rule may keep state of its own. Those come up in later chapters. For now the ledger says which values make up the model’s learned inventory.
See it move
Select W₁₂ in the output-layer view and change it from −1 to 0. Keep the input (1,1,0) and everything else fixed. The first weighted sum goes from 0 to 1 and the second stays 1. Reset before the next change, so every observation has one cause you can name.
Try changing a bias instead. Every input reaching that output gets the same added shift. Then switch the activation and leave the weights alone. The weighted sums stay put while the activation outputs move. The inspector shows the weighted sum and the activation output side by side.
A wiring diagram is good for following one contribution. A matrix table is better for seeing every parameter without crossing lines. Use whichever fits the question; the selected row and column stay attached to the same connection in both.
Where this shows up when you train
A saved parameter export is an inventory of numbers and shapes. The architecture is the recipe that says how to use them: which arrays multiply, where biases are added, which activation follows. Training changes the values and leaves that recipe fixed for the run.
A parameter edit now has a visible consequence. A cell changes a contribution, a row changes an output, and an activation changes the bend after the sum. The next question is which pieces of this recipe can be calculated at the same time, and which have to wait for earlier results.
What you now know
- Every dense connection is a multiplying weight in a matrix.
- Biases are learned parameters too.
- Changing the activation changes the recipe even if the weights stay fixed.
Where we’re headed
Next: which operations can run together, and which have to wait. Continue to the next chapter.