A model is a function
An engineer calls a language model “one large function.” What does that mean? We'll build small number machines, draw their behavior, and connect them. These few curves will keep returning throughout the book.
Functions are machines
A function applies a rule to an input. We read $f(x)$ as “f of x”: feed x into f, not f times x.
Input 3 → apply $f(x)=2x+1$ → output 7.
Worked example
$$ f(3) = 2 \cdot 3 + 1 = 7 $$In words: double the input $3$ to get $6$, add $1$, and the output is $7$.
Run the machine on a few inputs and record what it does. The table is the machine's behavior:
| input $x$ | rule $2x + 1$ | output $f(x)$ |
|---|---|---|
| $0$ | $2\cdot0 + 1$ | $1$ |
| $1$ | $2\cdot1 + 1$ | $3$ |
| $2$ | $2\cdot2 + 1$ | $5$ |
| $3$ | $2\cdot3 + 1$ | $7$ |
Check one row by hand. Pick $x = 2$, double it, add one, and confirm you also get $5$.
Composition connects machines. Let $g(x)=x^2$, meaning multiply x by itself. Read the innermost parentheses first:
Chain the machines in each order
$$ g(f(3)) = g(7) = 7^2 = 49 \qquad f(g(3)) = f(9) = 2 \cdot 9 + 1 = 19 $$In words: feeding $3$ through $f$ first and then $g$ gives $49$; feeding it through $g$ first and then $f$ gives $19$. Order matters, so you cannot swap the two machines around.
The result depends on order. Keep 49 in mind; we'll return to this chain when we trace slopes backward.
A graph is behavior you can see
A graph plots each input against its output. Our four pairs, (0,1), (1,3), (2,5), and (3,7), lie on a straight line.
The line adds two to its output for every extra input unit. The parabola $g(x)=x^2$ changes steepness: it falls on the left, flattens at zero, then rises. Inputs −2, −1, 0, 1, 2 give outputs 4, 1, 0, 1, 4. We can read that behavior before doing calculus.
$e^x$: growth by multiplying
An exponential grows by multiplying: $2^0=1$, $2^1=2$, $2^2=4$, $2^3=8$, and $2^{10}=1024$.
The constant $e \approx 2.718$ has a useful property: the slope of $e^x$ equals its height. We'll give “slope” a precise meaning next chapter.
$$ e^{x+1} = e \cdot e^x $$In words: adding $1$ to the input multiplies the output by $e$. Verify it by hand: $e^1 \approx 2.72$, and $2.72 \times 2.72 \approx 7.39 = e^2$ ✓.
| Input | Output |
|---|---|
| −2 | 0.14, approximately. |
| −1 | 0.37, approximately. |
| 0 | 1 exactly. |
| 1 | 2.72, approximately. |
| 2 | 7.39, approximately. |
For every finite input, the output stays positive. Large negative inputs produce tiny positive values, never negative ones.
$\ln x$: the question machine
The natural log, also called the natural logarithm, asks “e to what power gives x?” It is an inverse function. Thus $\ln1=0$, $\ln e=1$, and $\ln7.39\approx2$.
$$ e^{\ln x} = x \qquad \ln(e^x) = x $$In words: take the log of $x$ then feed it to $e^x$, and you are back at $x$; go the other way and the same thing happens. Each machine undoes the other.
For the first identity, x must be positive. The log has no real output at zero or below.
| Input | Natural log |
|---|---|
| 0.5 | About −0.69. |
| 0.1 | About −2.3. |
| 0.01 | About −4.6. |
Dividing the input by ten subtracts about 2.3 from the log. Negating the log will give a useful score for an unlikely correct answer.
$$ \ln(a \cdot b) = \ln a + \ln b $$In words: the log of a product is the sum of the logs. Hand-check it: $\ln 2 \approx 0.69$, and $\ln 4 = \ln(2 \cdot 2) \approx 1.39 \approx 0.69 + 0.69$ ✓.
This lets us store sums of logs instead of tiny products that a computer may round to zero, a numerical problem called underflow.
$\sigma(z)$: the squash
A sigmoid turns an unbounded score into a value between zero and one. Its symbol is $\sigma$, “sigma”; z names the input.
In words: take $e$ to the minus $z$, add $1$, and divide $1$ by the result.
At zero, $e^0=1$, so $\sigma(0)=1/(1+1)=0.5$.
Worked example
$$ \sigma(2) = \frac{1}{1 + e^{-2}} \approx \frac{1}{1 + 0.14} \approx \frac{1}{1.14} \approx 0.88 $$In words: $e^{-2} \approx 0.14$, add $1$ to get $1.14$, and $1$ divided by $1.14$ is about $0.88$.
The mirror-image input, $z = -2$, uses $e^{2} \approx 7.39$: $\sigma(-2) = \frac{1}{1 + 7.39} \approx 0.12$. The two are symmetric: $\sigma(2) + \sigma(-2) \approx 0.88 + 0.12 = 1$.
The curve is steepest at its middle and flattens toward either end. A different nonlinear rule, ReLU, clips negative inputs to zero. It bends without squashing positive outputs.
| Rule | Portrait | Where it helps |
|---|---|---|
| 2x+1 | Straight line. | Weight and added offset. |
| x² | Bowl. | Our first downhill walk. |
| eˣ | Rising exponential. | Make scores positive. |
| ln x | Slowly rising curve. | Score unlikely answers. |
| Sigmoid | S-curve. | Turn one score into a bounded share. |
| ReLU | Flat, then rising. | Add a bend between matrices. |
Play: one grapher, five personalities
Choose a curve, drag the input, and follow the point's output height. The static figures provide the same checkpoints.
See it move
Why the LLM cares
A language model composes many small functions. Matrices mix numbers; nonlinear functions bend the response. Exponentials will turn scores into positive quantities, and logarithms will measure prediction error.
Stacking straight-line machines still gives a straight-line machine. The bends between stages make richer behavior possible. A fixed model produces fixed scores for a fixed input; choosing a token from those scores can involve a separate random draw.
What you now know
- A function maps each allowed input to one output.
- Composition feeds one function into another; order matters.
- A graph shows how output changes with input.
- Exponentials multiply; logarithms undo them and turn products into sums.
- Sigmoid squashes; ReLU clips negative inputs.
Where we're headed
We can see where a curve is steep or flat. Next we'll measure its slope at one point and use that measurement to walk downhill.