The Classical Toolbox

The tools that still win

Deep learning gets the headlines, but walk into most companies solving most real problems — fraud on a spreadsheet, churn from a customer table, a diagnosis from lab values — and you will find, quietly winning, methods that predate deep learning by decades. This chapter is the field guide to those workhorses, and the honest case for when to reach for one instead of a neural network.

The methods split into the two jobs you already met in learning from data. Supervised methods learn from labeled examples — kNN, decision trees, forests, boosting, and support vector machines. Unsupervised methods find structure with no labels — k-means and PCA. And here is the thesis this whole chapter defends: on tabular data — rows and columns, like a spreadsheet — a good tree ensemble is usually the first thing to try and often the last, because neural networks need the kind of shaped data (images, text) a spreadsheet does not have.

To keep a survey of seven tools from turning into a shopping list, each gets three things and only three: its geometry (the picture of what it does to the space of data), its one key idea (usually a single equation), and its when (where it beats the alternatives).

k-Nearest Neighbors: ask your neighbors

Start with the tool that barely deserves the word "algorithm." To classify a new point, find the $k$ labeled points closest to it and take a majority vote. That is the whole thing — no training, no equation to fit; it just remembers every example and looks up the nearest ones when asked. Name it: k-nearest neighbors (kNN).

The geometry is nothing but distance. "Closest" means the straight-line distance between two points — the length of the vector between them, as in the math toolkit:

In words: subtract the two points coordinate by coordinate, square each difference, add them, and take the square root — the ordinary ruler distance.

The one knob that matters is $k$. Small $k$ (say $k = 1$) trusts the single nearest point, so the boundary turns jagged and chases every noisy outlier — an overfitting echo of the last chapter. Large $k$ averages over many neighbors, smoothing the boundary but blurring genuine detail. It is the same underfit-versus-overfit dial from last chapter in new clothes — the field's name for that trade-off is the bias-variance trade-off, which has nothing to do with a neuron's bias $b$ — and you set it on a validation set.

When does kNN win? On tiny datasets, boundaries with weird local shape, and as a dead-simple baseline you can write in five minutes. Its two curses, stated honestly: it must store and search all the training data at prediction time (slow and heavy at scale), and it falls apart in very high dimensions, where "nearest" stops meaning much because everything becomes roughly equidistant. Hold that second curse — PCA, at the end of the chapter, is one cure.

Decision trees: a game of twenty questions

The next tool plays twenty questions with the data. Ask a yes/no question about one feature — "is income > 50k?" — split the data into two groups, then ask another question of each group, and keep splitting until each group is nearly all one class. To predict, walk a new point down the questions to a leaf and read off that leaf's majority label. Name it: a decision tree.

The geometry is the payoff picture. Each question is a straight cut perpendicular to one axis: "income > 50k" is a vertical line, "age > 40" a horizontal one. So a tree chops the feature space into axis-aligned rectangular boxes, each carrying a single prediction. Unlike a neuron's single slanted line from the neuron, a tree can carve an intricate staircase of boxes.

The one idea that makes a tree work is how it chooses each split: it picks the question that best purifies the two groups — makes each side as close to single-class as possible. The common purity ruler is Gini impurity:

In words: $p_k$ is the fraction of class $k$ in a group, and the sum runs over the classes rather than over examples — same "add these up" sigma, new thing being counted. A group is pure (impurity 0) when one class fills it, and higher when it is more mixed. The tree greedily asks the question that makes its two child groups as pure as it can.

Why the squares? Because squaring rewards concentration: check it on two groups. A 50/50 group scores $1 - (0.5^2 + 0.5^2) = 1 - 0.5 = 0.5$, the most mixed two classes can be. A group that is all one class scores $1 - 1^2 = 0$, perfectly pure. Any lopsided group lands between them, so "smaller is purer" holds all the way down.

When does a tree win? It is the most interpretable model in the book — you can print the questions and a human can follow the reasoning, which matters enormously in medicine, credit, and law — and it handles mixed numeric-and-categorical features without fuss. But a single deep tree overfits ferociously: keep asking questions and every training point ends up alone in its own tiny box, which is memorization wearing a flowchart. The fix is not one tree but many — the next two methods.

Forests and boosting: many trees beat one

One tree overfits, but a crowd of trees, combined well, does not. There are two ways to build the crowd, and the difference matters, because one of them is, on tabular data, the champion of the whole field.

The first is the random forest — the parallel crowd. Grow hundreds of trees, each on a random subset of the data and a random subset of the features, so every tree overfits in its own direction. Then let them vote: the individual errors, being random and roughly independent, cancel out, and the shared signal survives. Name it: a random forest. It is an ensemble — the same averaging that made dropout work in teaching the network, done with whole trees.

The second is boosting — the sequential crowd, and the champion. Build the trees one after another, each new tree trained specifically to fix the mistakes the current crowd still makes. Tree one makes errors; tree two focuses on those; tree three on what is still wrong — each a small correction added to the running total. Name it: gradient boosting. The honest fact, plainly: gradient-boosted trees — the family behind XGBoost, LightGBM, and friends — win a large share of real-world tabular competitions and production systems.

The one distinction to carry away: a forest averages many independent guesses, taming the wild variance of single trees; boosting adds up many tiny corrections, chipping away at systematic error. Both generalize far better than one tree, and both keep more interpretability than a neural network — you can still ask which features the trees leaned on. When do they win? On tabular data, almost always, especially when you want strong accuracy without the data-hunger and tuning pain of a deep net.

Support vector machines: the widest street

When two classes can be separated by a straight boundary, there are infinitely many lines that do it — so which is best? The support vector machine picks the one that leaves the widest empty street between the classes: the boundary with the most breathing room on both sides. Name it: a support vector machine (SVM).

The geometry, and its one idea, is the margin. Draw the boundary, then push two parallel lines out from it, one toward each class, until they touch the nearest points; the gap between those touching lines is the margin, and the SVM maximizes it. The handful of points that touch the edges are the only ones that matter — the support vectors; move any other point and the boundary does not budge. For a boundary written $\mathbf{w} \cdot \mathbf{x} + b = 0$ — where $\mathbf{w}$ is the weight vector, drawn violet in the figure because it is a learned parameter — the street width is:

In words: the width of the street is two divided by the length of the weight vector, so maximizing the margin means making $\lVert \mathbf{w} \rVert$ small — the fewer, gentler the weights, the wider the street.

And if the classes are not linearly separable — a circle of one class inside a ring of the other? An SVM can secretly lift the points into a higher-dimensional space where they do separate by a plane, and the "kernel" computes distances in that space without ever building it. That is how a straight-line method draws curved boundaries — a different route to the curvature the neuron got from stacking hidden layers.

When does an SVM win? On medium-sized datasets with a clear margin, especially in high dimensions where the "only the support vectors matter" economy shines; text classification was an SVM stronghold for years. Its limits: it does not scale gracefully to millions of points, and its predictions are less interpretable than a tree's.

k-means: finding groups no one labeled

Now switch jobs — no labels at all (unsupervised). Given a cloud of points and no idea what the groups are, can we discover natural clusters? k-means says: guess there are $k$ clusters, and find them by a back-and-forth so simple you can run it by hand. Name it: k-means clustering.

The algorithm is Lloyd's two-step dance, and the widget below lets you drive it. Start: drop $k$ center points anywhere. Assign: color each data point by whichever center is nearest — the same distance ruler as kNN. Update: move each center to the average position of the points now assigned to it. Repeat, and the centers slide into the hearts of the clusters and stop. What it is secretly minimizing is the total within-cluster squared distance:

In words: add up, over every point $\mathbf{x}$ and its assigned center $\mathbf{c}$, the squared distance between them — make every point as close as possible to its own cluster's center.

The honest caveats, because k-means is easy to misuse. You must choose $k$ in advance — the algorithm will not tell you how many groups exist. The result depends on where you drop the initial centers, so bad luck gives a bad grouping (which is why it is run several times and the best kept). And it only finds roughly round, similar-sized blobs; it fails on long stringy or nested clusters. When does it win? On quick exploratory grouping, customer segmentation, and compressing the colors in an image — anywhere "find me some natural groups" is the whole ask.

PCA: squashing dimensions by shadow

The last tool, still unsupervised, answers a different need: data often has too many features (columns), most of them redundant or correlated. Can we describe it with fewer numbers without losing much? Name it: principal component analysis (PCA).

The geometry reuses the dot-product-as-shadow idea from the math toolkit — this is its big payoff. Picture a cloud of 2D points stretched into a long thin ellipse. Most of the spread runs along the ellipse's long axis; the short axis barely varies. PCA finds that long axis — the principal component — and describes each point by its shadow, its dot-product projection, onto that axis. Throw away the short-axis coordinate and you have turned 2D into 1D, losing almost nothing, because the short axis held almost no information.

The one idea, without heavy machinery: PCA seeks the direction of maximum variance — the most spread-out direction — then the most-spread direction perpendicular to that, and so on, ranked axes of the data's own shape. Keep the top few, drop the rest. The coordinate of a point along a component is exactly the dot product you already own:

In words: a point's position along a component is just its shadow on that direction — the dot product from the math toolkit, doing dimension-reduction.

One clause of honesty about where the axes come from: the components are computed from the data's covariance, and that machinery (eigenvectors of the covariance matrix) is beyond us here — but the picture, long axis and short axis, is exact. When does PCA win? On squashing hundreds of correlated features down to a handful before feeding another model (curing kNN's high-dimension curse), on visualizing high-dimensional data in 2D, and on compression. Its honest limit: it only finds straight axes of variation, so genuinely curved structure escapes it — for which nonlinear cousins like t-SNE and UMAP exist, mostly for visualization.

Where you'll meet this

Pull the whole survey into a single decision. Do you have labels? No — reach for k-means or PCA to explore. Yes — then is the data a spreadsheet of rows and columns? Then start with gradient-boosted trees; they will likely win, train in minutes, and tell you which features mattered. Is the data an image, a sentence, or a graph — data with shape? Then, and mostly only then, reach for the deep networks the next part builds.

State the engineering truth this chapter defends, once and cleanly: a neural network is not the best tool — it is the best tool for data with structure a network can exploit. On a mid-sized tabular dataset, a boosted-tree ensemble usually beats a neural net on accuracy, training time, and interpretability, which is why it remains the professional's default there. Choosing deep learning where a tree would win is one of the most common and expensive mistakes in the field.

Here is precisely why images and text change that answer. Their structure — a pixel's neighbors, a word's order, a node's links — carries the signal, and a plain network (or a classical method) throws that structure away by treating every feature as unrelated. The next chapters wire networks that preserve and exploit shape: convolution for grids, recurrence and attention for sequences, message-passing for graphs. That is where deep learning earns its headline. You now hold a genuine toolbox — and, more valuable than any one tool, the judgement to choose; the capstone chapter, choosing your tool, turns that judgement into a full field guide.