Part III · Ch. 12 — Learning by Doing

Part III · Chapter 12 of 13

Learning by Doing

Reinforcement learning in one honest chapter

You’ll need:the three ways a machine learns — reinforcement learning was named as the third, learning by trial and reward; this chapter opens it upprobability and expectation (an average outcome), and derivatives as 'which way is downhill'a neural network trained by gradient descent — here, a network becomes the agent's decision-maker

Learning by consequence

Nobody handed a computer the strategy for the game of Go and then watched it win. The machine started knowing only the rules, played millions of games against itself, and kept whatever led to winning — until it beat the best human on Earth with moves that stunned the champions. There were no labeled examples of "good move here." Just moves, outcomes, and adjustment. That is a completely different way to learn, and it is the one we have named twice and never opened.

Back in learning from data we drew three ways a machine can learn: supervised (examples that come with answers), unsupervised (structure with no answers), and reinforcement — learning by trial and reward. Everything since has been the first two. This chapter is the honest, single-chapter tour of the third: reinforcement learning (RL), where there is no dataset at all — only a world to act in and consequences to learn from.

Ground it in something everyone knows before any diagram: training a puppy. You do not hand the puppy a labeled list of correct behaviors. It tries things; good things earn a treat; over time it does more of what gets treats. The puppy is an agent, your kitchen is its environment, sitting-on-command is an action, and the treat is a reward. Reinforcement learning is that loop, made precise — so let us make it precise.

The loop: state, action, reward

Two things exist, and the whole field beats around the rhythm between them: the agent (the learner) and the environment (the world it acts in). They talk in a loop that never breaks its rhythm.

A vertical loop between two boxes. The top box, outlined in mint, reads 'AGENT (the learner)'. The bottom box, outlined in muted gray, reads 'ENVIRONMENT (the world)'. Down the right side a mint arrow points from the agent to the environment, labeled 'action a'. Up the left side a single arrow points from the environment back to the agent, carrying two stacked labels: 'state s' in blue and 'reward r' in amber. In the center of the loop a muted italic note reads 'no labels — the reward is the only teacher'. Below the whole diagram a muted strip reads 'the agent chooses; the world answers with a new state and a number; repeat forever'.
The reinforcement-learning loop. The agent takes an action; the environment answers with a new state and a single reward number; the agent adjusts and goes again. That reward is the entire teaching signal — no one ever tells the agent the correct action, which is exactly what separates RL from supervised learning.

Walk one turn. The environment shows the agent a state (where things stand right now). The agent picks an action. The environment responds with two things: a new state, and a reward — a single number, its only feedback. Then it repeats: state, action, reward, and around again. Notice the starkness: that reward is the entire teaching signal. No one ever says "the right action was $X$." The agent has to work that out from a running stream of scores.

The agent's real goal is subtler than "get a reward." It is not chasing the next reward — it is chasing the largest total reward over the whole future. A chess agent will happily sacrifice a piece (a small immediate loss) for checkmate later. So the quantity that matters is the sum of all rewards from here to the end, and that raises the central difficulty of the entire field, the one the rest of this chapter circles: how do you learn to act now for a payoff that comes much later? One thing more sets RL apart — unlike supervised learning's fixed dataset, the agent generates its own data by acting, which is powerful (it can explore situations no one thought to label) and treacherous (its early bad choices poison the very experience it later learns from).

Policy and value: a plan, and a price tag on every square

Two ideas are what the agent is actually learning. The first is the policy, written $\pi$ — the Greek letter pi, here meaning "the agent's strategy," nothing to do with $3.14$. A policy is a rule from states to actions: in this situation, do that. Learning to act well is learning a good policy. Early on the policy is random flailing; a trained policy is skill.

The second idea is how the agent copes with delayed reward — the value of a state: a price tag saying "starting from here and playing on, how much total reward do I expect to collect?" A state one move from checkmate is high-value even if standing there earns nothing right now, because of what comes next. Value is how a far-off reward reaches back and makes a present state look good.

Make it concrete with a tiny worked world. Picture a corridor of four squares in a row: $s_1$, $s_2$, $s_3$, and a goal. The policy is "always step right." Every step earns $0$, except the step into the goal, which earns $+1$, and then the episode ends. Before we compute anything, one new idea: a reward later is worth a little less than the same reward now, so we shrink each future step by a factor $\gamma$ — the Greek letter gamma, the discount factor — here $\gamma = 0.9$. Discounting is sane (a treat today beats the identical treat next year) and convenient (it keeps the running total finite).

A horizontal corridor of four equal squares. The first three, outlined in gray, are labeled inside 's1', 's2', 's3'. The fourth is faintly amber with an amber border, labeled 'GOAL' with a smaller '+1 reward'. A small mint agent dot sits in the first square, labeled 'agent'. Between consecutive squares, small mint arrows point right, each labeled 'step (0)', except the arrow into the goal which is labeled '+1' in amber. Below each square a violet value tag reads, left to right, 'V = 0.81', 'V = 0.90', 'V = 1.00', and 'V = 0' under the goal. Above the row a muted dashed arrow arcs from right to left, labeled 'value flows backward from the reward, dimming by times 0.9 each step (gamma = 0.9)'.
Value flows backward from reward. In this corridor the only reward ($+1$) sits in the goal, yet with a discount of $\gamma = 0.9$ every earlier square inherits a dimmed share of it — $1.00$, $0.90$, $0.81$ — because value asks "how much total reward do I expect from here on?" Those numbers are a gradient the agent can climb toward the goal, even though the reward itself lives in just one square. Each is the next square's value times $0.9$.

Two small pieces of notation make this exact. The total reward the agent is really chasing, counting the whole future but discounting each later step, is the discounted return:

$$G = r_1 + \gamma\, r_2 + \gamma^2 r_3 + \cdots$$

In words: add up all the future rewards, but shrink each later one by another factor of $\gamma$ — the reward one step away counts fully, two steps away counts $\gamma$ times as much, three steps away $\gamma^2$ times, so "sooner" is worth more than "later." Here $r_1$ is the reward from the next step, $r_2$ from the step after, and "$+\cdots$" just means "keep going to the end."

The value of a state obeys a neat one-step relation. Writing $s'$ (read the little prime as "the next one") for wherever the agent lands after acting:

$$V(s) = r + \gamma\, V(s')$$

In words: a state's value is the reward you get now plus the discounted value of wherever you land next. You do not have to sum the whole future by hand every time — each state just borrows from its neighbor.

Worked example — value flows backward down the corridor

Compute the values right to left, so you feel the reward propagate. From $s_3$ the very next step reaches the goal and collects the $+1$, so its value is that reward with nothing after it:

$$V(s_3) = 1.00$$

From $s_2$ the goal is two steps off. The immediate reward for stepping is $0$, so the value is purely the discounted value of the next square:

$$V(s_2) = \gamma \times V(s_3) = 0.9 \times 1.00 = 0.90$$

From $s_1$, one square further back, discount again:

$$V(s_1) = \gamma \times V(s_2) = 0.9 \times 0.90 = 0.81$$

In words: the picture to leave with is $0.81, 0.90, 1.00$ — value dimming by a factor of $0.9$ with every step of distance from the goal. The reward itself lives in exactly one square, but its influence bleeds backward into every square that leads there. (The goal is terminal, so its own value is $V = 0$: there is no future left once you arrive.)

Here is the deep point this unlocks: if you know the value of every state, acting well is easy — from wherever you are, step toward the higher-value neighbor. So a huge part of RL is just learning good value estimates. And the network from teaching the network is exactly what learns them: a network that eats a state and outputs its value, trained by gradient descent, with the reward stream providing the target.

Why it's genuinely hard

Reinforcement learning is the most powerful and the most temperamental of the three ways to learn, and it is worth knowing why before you reach for it. There are two difficulties that supervised learning never has to face.

Two panels split by a vertical hairline. The left panel is headed 'credit assignment — who won it?': a left-to-right chain of six small circles joined by thin arrows, ending in an amber star labeled 'win! +1'. Above the chain a curved amber dashed arrow sweeps back over all the moves, labeled 'which of these earned it?'. One middle circle is tagged in mint 'great move' and another in red 'blunder (got away with)'. A note reads 'one reward, many past actions'. The right panel is headed 'explore vs exploit': a mint agent dot faces two arrows — a solid one to a blue box 'known: +2 (reliable)' and a dashed one to a muted box 'untried: +? (maybe better, maybe worse)'. A note reads 'take the sure thing, or gamble on something better?'. A bottom italic line reads 'delayed reward and the explore/exploit dilemma are what supervised learning never has to face'.
Two difficulties supervised learning never faces. Left, credit assignment: a single reward arrives at the end of a long chain of actions, and the agent must work out which moves actually earned it and which were dead weight. Right, explore versus exploit: every turn the agent must choose between the reliable known payoff and an untried action that might be better — or worse. Balancing these is the heart of what makes reinforcement learning hard.

Difficulty one is the credit assignment problem — delayed reward. You win a 40-move chess game. Which move won it? The reward ($+1$) arrives only at the very end, but it must somehow reward the good move on turn 12 and not the blunder on turn 30 that you got away with. Spreading a single delayed reward back over the long chain of actions that earned it — figuring out who deserves the credit and who was just along for the ride — is the central technical problem of RL. And it is exactly what the discounted value from the last section is quietly solving.

Difficulty two is exploration versus exploitation. The agent only learns about actions it actually tries. So every turn it faces a dilemma: exploit what it already knows works (take the reliable reward), or explore something untried that might be better (and might be worse). Lean too hard on exploiting and it locks into the first mediocre strategy it stumbles on, never discovering the great one; explore too much and it never cashes in what it learned. Every RL system must balance the two — a tension with no clean answer, only tunable tradeoffs. The plainest image is a restaurant: always ordering your usual dish (exploit) versus trying a new one (explore) — the only way to find a better favorite, and the only way to get a worse dinner.

The honest headline, so nobody over-reaches for RL: because the agent generates its own data by trial and error, RL is hungry (it can take millions of attempts), unstable (a small change can send learning off a cliff), and often needs a safe simulator to fail in — you cannot crash ten thousand real cars to train a driver. It is spectacular when it fits and painful when it does not, which is the whole reason the last chapter of this book is about choosing.

How it actually learns

Here is the reassuring part: the engine is the one you already own. Nothing new under the hood. The agent's policy (and its value estimates) are just networks with adjustable weights, and they are trained by the same gradient descent from fitting a line and teaching the network. What changes is only where the learning signal comes from.

The one-line intuition of policy learning, no policy-gradient math required: try an action; see how much reward followed; if it was more than expected, nudge the weights to make that action more likely in that state next time; if less, nudge them to make it less likely. It is gradient descent, but the "loss" is now built from the reward stream instead of from a fixed set of correct answers — the reward has taken the place of the labeled target. And that word "expected" is where value locks back in: because a raw reward is delayed and noisy, agents usually lean on their learned value estimates as a stand-in target — "was this action better than my value estimate said it would be?" — which is how the credit-assignment problem gets tamed in practice.

So a curious reader has handholds, here is the landscape in one clause each. There are value-based methods (learn the value of actions, then act greedily — Q-learning is the classic), policy-based methods (adjust the policy directly), and the actor-critic hybrids that pair the two and power most modern systems. The takeaway is the shape, not the recipe: RL is gradient descent pointed at a reward signal, coping with delay through value.

Where you'll meet this

Now the payoff that ties both sites together — reinforcement learning is not exotic, it is quietly finishing the chatbot you use. Recall from the sister LLM site that a raw language model is a next-token predictor trained on a mountain of text (the sister site's prediction game); helpful, harmless, honest behavior is not something plain text-prediction gives you. The polish that turns the parrot into an assistant is reinforcement learning.

A loop drawn with RL terms mapped onto a language model. A top box outlined in mint reads 'the LLM = the AGENT' with a small subnote 'policy = how it responds'. A mint down-arrow labeled 'action = write a response' leads to a small speech bubble tagged 'a candidate answer'. That answer feeds right into a box outlined in amber reading 'reward model', with a subnote 'learned from humans ranking answers'. An amber up-left arrow returns from the reward model to the LLM, labeled 'reward = preference score'. To the right of the reward model, a small cluster shows two stacked answer chips with 'A > B' between them and a tiny human glyph, labeled 'people ranked pairs, trained this'. In the center a muted italic note reads 'same loop as the puppy and the treat — at trillion-parameter scale'.
The assistant you chat with was finished by reinforcement learning. In RLHF the LLM is the agent, writing a response is its action, and the reward comes from a second model that learned what people prefer (by having humans rank pairs of answers). Gradient descent nudges the LLM toward higher-scoring responses — the very loop from earlier in this chapter, run at the scale of a giant model. Its helpful manners are a trained policy, not a fact it memorized.

Walk the loop in the vocabulary we just built. It has a name: reinforcement learning from human feedback (RLHF). The LLM is the agent; writing a response is its action; the state is the conversation so far; and the reward comes from a second model that learned what humans prefer — people ranked pairs of answers, that trained a reward model, and the reward model now scores every response. Gradient descent then nudges the LLM's policy toward responses that score higher: the same loop as the puppy and the treat, at the scale of a trillion-parameter model. The sister site's from parrot to partner tells this story from the LLM side (and notes a reinforcement-learning-style shortcut, DPO, that rewards preferred responses without a separate reward-model loop). The assistant's manners are a trained policy.

RL is no one-trick idea. It is how machines reached superhuman play at Go, chess, and video games, how robots learn to walk and grasp, how data centers and chip layouts get tuned, and how recommendation systems decide what to show you next. Wherever the goal is a sequence of decisions judged by an outcome rather than a single labeled answer, RL is the tool.

You now hold all three ways to learn and every major family of model — from a line-fitter to a transformer, from a random forest to a diffusion model, and now an agent that learns by doing. The one skill left is the professional's real job: given a fresh problem, looking at it and choosing the right tool. That is the capstone, next.

What you now know

  • Reinforcement learning is the third way a machine learns: an agent takes actions in an environment and receives a reward — a single number that is its only feedback — and adjusts to earn more total reward over time, with no labeled examples of the right action anywhere.
  • The agent chases the largest total future reward, not the next one, which is why it will accept a small immediate loss for a bigger later payoff; the discount factor $\gamma$ shrinks each future reward so the total stays finite and "sooner" beats "later."
  • A policy $\pi$ is the agent's strategy (which action in each state); the value of a state is the total future reward expected from it — and in the worked corridor, value flows backward from the $+1$ goal as $1.00$, $0.90$, $0.81$, each the next square's value times $\gamma = 0.9$.
  • RL is hard for two reasons supervised learning never faces: credit assignment (a delayed reward must be shared out over a long chain of past actions) and the explore-versus-exploit dilemma (use what works, or gamble on something possibly better).
  • Under the hood it is still gradient descent — the same engine as line-fitting — but the learning signal is built from the reward stream instead of labeled answers: make reward-better-than-expected actions more likely, worse ones less likely.
  • It runs quietly in the tools you use: RLHF turns a raw next-token predictor into a helpful assistant by making the LLM the agent, a response the action, and a human-preference model the reward — the same loop as a puppy learning tricks, at enormous scale.

Where we're headed. You now hold the whole kit. Three ways to learn — supervised, unsupervised, and, as of this chapter, reinforcement — and every major family of model built on top of them: a line-fitter and a logistic classifier, a neuron and a deep network, the classical toolbox of trees and forests and clustering, convolutional networks that see, recurrent and attention-based networks that read sequences, graph networks that reason over connections, generative models that make things up, and now agents that learn by doing. What is left is the skill none of those chapters could teach on its own, because it lives above all of them: given a real problem you have never seen, how do you look at it and choose the right tool — and defend the choice? That judgment is the whole point of the field, and it is the capstone. Let's put the toolbox to work.