Learning by Doing

Learning by consequence

Learning Go through self-play means starting with the game’s rules, playing millions of games, and adjusting toward moves that lead to wins. The rules tell you what is legal; they do not supply a labeled “best move here.” The strategy has to come from the consequences.

In learning from data we named three families. Reinforcement learning (RL) is the one where a learner acts and uses rewards to improve its behavior. It does collect data, namely experience from those actions, but the feedback need not say which action was correct.

The difference is what the feedback tells us. A labeled example names the answer to match. A reward scores what happened after a choice, often leaving us uncertain which earlier choice deserves credit. The learner must connect the outcome to its behavior across time.

Training a puppy gives us the picture. It tries things, sitting on command earns a treat, and over time it does more of what earns treats. The puppy is the learner, your kitchen is the world it acts in, sitting is a choice, and the treat supplies feedback. The rest of the chapter makes that loop precise.

1. Try a behavior. The puppy sits.
2. Receive feedback. Sitting earns a treat.
3. Learn from it. Rewarded behavior becomes more likely.

The loop: state, action, reward

The agent chooses actions. The environment is everything outside it: the world that responds. A game, simulator, or physical system can play that role.

A state describes the current situation. An action is a choice that can change it. A reward is the number scoring the outcome. The agent receives the new situation as well as the reward; the reward is its objective’s teaching signal, not a label naming the correct move.

1. Observe. Read the state: the board position or current measurements.
2. Choose. Take an action using the current strategy.
3. Receive. Record the next state and reward from the environment.
4. Repeat. Use experience to improve the strategy over training, then act again.

Learning need not happen after every individual action. An agent may gather a whole episode or reuse stored transitions before updating its rule. “Repeat” describes interaction with the environment; adjusting weights is a training operation using that experience. Keeping those two rhythms separate helps when a reward arrives only after many choices.

The goal is the largest total future reward, not the next reward alone. A chess agent can sacrifice a piece for checkmate later. If a score is delayed, acting now for a later payoff requires learning how present choices influence later situations. One favorable immediate outcome is not enough to tell us an action was wise.

That cuts both ways. RL can try situations nobody thought to label, and bad early choices can restrict the very experience it learns from. Stored experience can be reused, so “learning by doing” does not mean throwing away every past attempt.

Policy and value: a plan, and what each square is worth

A policy, written $\pi$ (Greek pi, meaning strategy, nothing to do with $3.14$), maps states to actions. A state’s value estimates future reward under that policy, even when standing there earns nothing.

Take four corridor squares, $s_1$, $s_2$, $s_3$, and goal. Our policy always steps right. Steps pay $0$ except entering the goal, which pays $+1$ and ends the episode. The discount factor $\gamma$, Greek gamma, is $0.9$: near enough to one to preserve the goal’s influence, and easy to do by hand.

Discounting values sooner rewards more and keeps infinite sums of bounded rewards finite when the factor is below one. The return $G$ adds rewards $r_1$, $r_2$, and so on, starting with the next step:

In words: add all future rewards, shrinking each later one by another factor of gamma. The next reward $r_1$ counts fully; $r_2$ counts $0.9$ times as much; the third counts $0.9^2$. The dots mean keep going to the end: sooner rewards count more than later ones.

Write value as $V(s)$. For this deterministic corridor and fixed policy, let $s'$ (“s-prime”) mean the next state and $r$ the immediate reward:

In words: a state’s value is the reward you get now plus the discounted value of wherever you land next. Each state borrows from its neighbor, so we do not have to add the whole future again.

Compare actions by immediate reward plus discounted next-state value; choosing the highest-valued neighbor alone fails at the terminal goal. With uncertain transitions, value averages over possible outcomes.

Why it's hard

RL must learn from outcomes while choosing which experiences to collect. Four difficulties shape that work.

These problems interact. An agent that never explores lacks the evidence for better value estimates, and a misspecified reward still produces the wrong behavior however well it is optimized. Reach for RL when sequential consequences require it and you have a practical way to collect useful experience.

How the learning works

In deep RL, the policy and value estimates are networks with adjustable weights. The optimizer is familiar from fitting a line and teaching the network; the learning signal now comes from experience and rewards. A value network takes a state as input and predicts its value. A policy network chooses actions, often by assigning each a probability.

1. Try. Choose an action and collect what follows.
2. Compare. Was its outcome better or worse than expected?
3. Adjust. Make better-than-expected actions more likely in that state, and worse ones less likely.

That is the intuition behind policy learning, not a complete algorithm. Rewards are delayed and noisy, so a learned value estimate helps judge what was expected. The loss uses that comparison to guide changes to the policy’s weights. Value learning, in turn, uses rewards and estimates of what comes next to improve its predictions.

A lucky outcome proves nothing about the actions that preceded it. Repeated experience helps separate a useful choice from a blunder that happened to be followed by success. Comparing against an expectation asks whether the outcome was better than the agent had reason to predict, instead of treating every positive score as equally informative.

Small problems can store values in a table rather than train a network. RL names the learning problem; gradient descent is one way to fit its functions. In either case, the difficult part is making useful updates from a stream of consequences instead of a fixed answer key.

From reward to an assistant’s behavior

A raw language model learns next-token prediction from text, which is the sister site’s prediction game. That objective alone does not specify the helpful behavior we want in a conversation. One way to shape it is reinforcement learning from human feedback, or RLHF.

1. Rank. People compare pairs of candidate answers.
2. Fit a judge. Train a reward model to predict those preferences.
3. Adjust the agent. Score generated responses, then train the language model toward higher-scoring behavior.

The reward model estimates preferences; its score does not guarantee truth or usefulness.

At enormous model scale, this is still the same action-and-feedback loop. The sister site’s from parrot to partner develops the story and covers direct preference optimization (DPO), a method that learns from preferred responses without a separate reward-model RL loop. Preference training can shape a model’s behavior through more than one recipe.

Where you'll meet this