Learning from Data

Two ways to teach a computer

Imagine you want a computer to sort your incoming mail into "spam" and "not spam." The old way: you sit down and write the rules — if it says "free money," flag it; if it comes from grandma, keep it. You quickly discover there are ten thousand rules, they contradict each other, and the spammers change tactics every week. There is a second way, and it is the whole subject of this site.

Instead of writing the rules yourself, you show the machine thousands of emails that a human has already labeled "spam" or "not spam," and you let the machine find the rule. That flip — from writing rules to finding them — is what the phrase machine learning means, and it is why this site exists.

Here is the frame for everything that follows. I'm going to build this field up from nothing, one honest idea at a time, until you could look at a real problem and pick the right tool for it — no hand-waving, no "it just works." This first chapter is the map; the rest of the site is the territory.

And you need no math background yet. This chapter is entirely in words and pictures — the math toolkit comes next chapter, exactly when we need it and not a moment before.

The flip

Picture it before we put it in words.

Classical programming is a pipeline: rules and data go in, answers come out — and you wrote the rules by hand. Machine learning turns the pipeline around. Data and answers go in, and the rules come out. Then you run those found rules on new data to get new answers.

On the spam example: the classical way, you write "if it contains X, then spam" and feed it emails, and out come labels. The machine-learning way, you feed it emails and their known labels, and out comes a rule-finder that has worked out, on its own, what spam tends to look like.

Why does the flip win where it wins? Some tasks have rules too numerous, too fuzzy, or too shifty to write down. Nobody can write the rule for "this photo contains a cat" — try it, you will fail — but a child learns it from a few hundred examples, and so can a machine. When the rule is easy to state, write it yourself. Machine learning earns its keep exactly when the rule is easy to demonstrate but hard to describe.

If "finding a rule" is the job, it helps to say precisely what a "rule" is. It turns out every rule in this book is the same kind of object — a function.

Every model is a function

Here is the single frame that unifies the whole field. A function is a machine that takes an input and returns an output. We will write it $f$: you feed in $x$, it hands back $f(x)$. Same input, same output, every time.

One box, labeled $f$, with three jobs passing through it: a house's size in, a price out; an email in, "spam" or "not spam" out; a photo in, "cat" or "dog" out. Different problems, identical shape.

Now the thesis of this entire site, stated plainly. Every machine-learning model in this book — linear regression, a decision tree, a neural network, the thing behind your favorite chatbot — is a function $f$, and "training" is the search for the settings of $f$ that make its outputs match reality. Learning is function-fitting. The rest of this site is filling in what $f$ can look like and how the fitting is done.

The field has two words for the two halves of a function's job, and it is worth meeting them now so they feel familiar when they arrive. When the output is a quantity — a price, a temperature, a number on a dial — the task is called regression. When the output is a category — spam or not, cat or dog, which of ten digits — the task is called classification. Chapter 3 does our first regression; Chapter 4 does our first classification.

Three ways a machine learns

The flip needs "answers" to learn from — but there are three genuinely different situations, depending on what answers you have. They are the three great families of machine learning.

Supervised learning is learning from labeled examples. You have inputs and their correct answers: emails with "spam / not-spam" tags, houses with their sale prices, photos with "cat / dog" captions. The machine fits a function so its outputs match the labels, then generalizes to new inputs. This is the workhorse — most of this site, and Chapters 3 and 4 specifically, live here. Supervised learning is learning with an answer key.

Unsupervised learning is finding structure with no answer key. You have inputs only, no labels, and you ask the machine to find the natural groupings or hidden axes on its own: which customers behave alike, which genes switch on together, what the main directions of variation are. Two moves to name: clustering (grouping similar things — the k-means method in Chapter 8) and dimensionality reduction (finding the few directions that matter). Unsupervised learning is finding the pattern nobody labeled.

Reinforcement learning is learning from consequences. There is no fixed answer key at all; instead an agent tries actions in a world and gets rewards or penalties, and it learns a strategy that earns more reward over time — how a program learns to play a game, steer a robot, or (in Chapter 12) how the polish on a chatbot's manners gets trained. Reinforcement learning is learning from trial, error, and reward.

Land the distinction so it sticks: supervised — someone tells you the right answer; unsupervised — nobody tells you anything, find the shape yourself; reinforcement — nobody tells you the right answer, but the world tells you afterward whether it went well.

Almost everything in Parts I and II of this site is supervised learning, because it is the clearest place to learn how fitting works — but keep the other two in mind. The capstone chapter's whole job is choosing among them for a real problem.

AI, ML, deep learning: which is inside which

Three words get used interchangeably, and they shouldn't be. They are nested, like Russian dolls.

Artificial intelligence — the outer ring — is the oldest and broadest goal: get machines to do things that seem to require intelligence. It includes machine learning, but also older ideas that do not learn from data at all: hand-written expert rules, search algorithms, game trees. A chess engine from the 1990s was AI without being machine learning.

Machine learning — the middle ring — is the subset of AI that learns its rules from data. That is the flip from two sections ago. Everything on this site from Chapter 3 onward lives in this ring.

Deep learning — the inner ring — is the subset of machine learning that uses neural networks with many layers stacked up. It is the technique behind Part II of this site and behind essentially every headline AI system of the last decade. Deep learning is machine learning; machine learning is AI; but the arrows only point outward.

The takeaway you can now hear in the wild: when a news article says "AI," it almost always means a deep-learning model — the innermost doll — even though the word it uses is the outermost. Now you can tell the difference.

What "training data" really is

Let's get concrete about the fuel. For supervised learning, the training set is nothing more mysterious than a table: each row is one example, with the input in some columns and the correct answer in the last column. Ten thousand emails, each with its words and its human-assigned "spam / not-spam" label. Throughout this site we will call it "our $N$ examples" — $N$ is just how many rows we have.

The one iron law of training data: the model can only be as good as the examples it learns from. Feed it biased examples and it learns the bias; feed it mislabeled examples and it learns the mistakes; show it only sunny-day photos and it fails in the rain. "Garbage in, garbage out" is not a slogan here, it is a theorem. Much of the real work in machine learning is not clever algorithms — it is getting good data.

The other half of the iron law: the model is graded on data it has never seen. We hold some examples back, hide their answers, and check whether the found rule generalizes. A rule that aces the examples it studied but flops on new ones has overfit — it memorized the answer key instead of learning the subject. This tension between memorizing and generalizing is the quiet villain of the whole field. Name it now; we meet it properly in Part II.

This quietly explains something you have probably noticed: the reason a model that has read much of the internet can still be confidently wrong is written right here. It learned from what people actually wrote, warts and all, and generalizes from that — not from some separate reservoir of truth.

Where you'll meet this

Let me turn the map into a route through the site. Everything ahead is one of two moves: choosing what shape the function $f$ can take, and choosing how to fit it to data. That is the spine every later chapter hangs on.

Here is the itinerary. Next chapter assembles the five mathematical tools every model uses — vectors, dot products, matrices, slopes, and probability. Chapter 3 fits the simplest possible function, a straight line, and you will watch a machine "learn" for the first time. Chapter 4 bends that line into a decision boundary and does our first classification. Part II stacks these pieces into neural networks. Part III matches network shapes to the shape of the data — images, sequences, graphs. And the final chapter puts a real problem in front of you and asks you to choose the tool.

You now own the vocabulary: supervised versus unsupervised versus reinforcement, regression versus classification, AI versus machine learning versus deep learning, training data and overfitting. Every later chapter is a deeper look at a corner of this one map. When you feel lost, come back here — this is the picture the details hang on.

And to close on the promise this site opened with: I built this so a friend with zero math background could get the real answer, not a cartoon of it. The real answer starts with the sentence at the top of this chapter — machine learning is finding the rule instead of writing it — and it does not require a single equation to be true. The equations, starting next chapter, are only there to make it precise.