The tools that still win
Deep learning gets the headlines, but a customer table, fraud spreadsheet, or instrument log is often a good place to try classical methods first. Seven workhorses cover most of that work, and each wins under different conditions. A neural network should earn its complexity against a suitable baseline.
The jobs are the supervised and unsupervised ones we already know. With labels, we predict answers; without them, we explore structure. On tabular data, meaning rows and columns like a spreadsheet, a tree ensemble is a strong starting point and often enough for the job. Treat that as a default worth testing; networks can learn from tables too.
| Job | Tools | What you get |
|---|---|---|
| Predict a label | kNN, tree, forest, boosting, SVM | An answer for a new example |
| Explore without labels | k-means | A chosen number of groups |
| Reduce dimensions | PCA | Coordinates along the directions of greatest spread |
Training cost and prediction cost are different questions. A neighbor method can be ready almost immediately yet do substantial work for every new query. A fitted tree spends effort choosing splits, then predicts by following a short path. Ask both questions when you compare methods.
| Before choosing | Ask |
|---|---|
| Labels | Are there example answers, or only inputs? |
| Geometry | Does distance, a sequence of cuts, or a projection preserve useful structure? |
| Explanation | Must we follow each decision, or is aggregate feature importance enough? |
| Evaluation | Does it beat a simple baseline on examples held out from fitting? |
k-Nearest Neighbors: ask your neighbors
To label a new point, find its $k$ closest labeled examples and take their majority vote. k-nearest neighbors (kNN) stores examples rather than fitting weights.
| When it wins | Small datasets and local boundaries; quick baseline. |
|---|---|
| When it loses | Storage, search cost, or unhelpful high-dimensional distances. |
We choose three neighbors so the vote includes more than one example and cannot tie between two classes. Three is a teaching choice here, and real work tunes it. The distance rule is also a choice: two records count as similar only if their coordinates and units make ruler distance meaningful. Stored examples supply the answer, but our representation determines which examples get a vote.
Let $\mathbf{x}$ be the query and $\mathbf{x}_i$ a stored point. Their ruler distance is the vector length from the toolkit:
$$d(\mathbf{x}, \mathbf{x}_i) = \lVert \mathbf{x} - \mathbf{x}_i \rVert = \sqrt{(x_1 - x_{i1})^2 + (x_2 - x_{i2})^2}$$In words: subtract coordinates, square the differences, add them, then take the square root.
| Distance recipe | Result |
|---|---|
| Coordinate differences → squares → sum → root | Ruler distance |
Worked example: one vote
Our query is $\mathbf{x}=(4,5)$. Calculate distances to two stored points:
$$d\big((4,5),\,(5,5)\big) = \sqrt{1^2 + 0^2} = 1.0 \;\;(\text{B}) \qquad d\big((4,5),\,(2,4)\big) = \sqrt{2^2 + 1^2} = \sqrt{5} \approx 2.24 \;\;(\text{A})$$The other nearby B point, $(6,5)$, is $\sqrt{2^2+0^2}=2.0$ away, closer than A at $2.24$.
In words: sorted distances are $1.0$ (B), $2.0$ (B), $2.24$ (A); $k=3$ gives a 2-to-1 vote for B.
| Neighbor rank | Label |
|---|---|
| First and second | B, B |
| Third | A |
Choose $k$ on validation data. The bias-variance tradeoff is unrelated to a neuron’s bias $b$:
| Neighborhood | Risk |
|---|---|
| $k=1$ | One noisy example decides |
| Large $k$ | Real local detail gets blurred |
Scale features before measuring distance: a column measured in thousands can otherwise dominate one measured in single units. In many dimensions, distances can become less distinctive; PCA below can help when redundant dimensions obscure the structure.
Decision trees: a sequence of questions
A decision tree asks questions such as “income > 50k?” and keeps splitting groups. Prediction follows the questions to a leaf and uses its majority class.
| When it wins | Small, readable decision rules on tables. |
|---|---|
| When it loses | Deep trees memorize; small data changes can alter splits. |
Read the pictured tree as a procedure. If $x_1>4$, predict B immediately; otherwise test $x_2>3$, predicting B above that cutoff and A below or on it. The second cut applies only to the left branch. This is why the horizontal line stops at the vertical one instead of dividing the entire plane.
| First cut | Second cut |
|---|---|
| Whole plane | Left branch only |
Each single-feature question makes an axis-aligned cut: income makes a vertical cut, age > 40 a horizontal one. The tree partitions the plane into rectangular regions, unlike the slanted boundary of a neuron.
A tree chooses splits that make labels less mixed, measured here by Gini impurity:
$$\text{impurity} = 1 - \sum_k p_k^2$$In words: $p_k$ is a class’s fraction in the group; sum squared fractions over classes and subtract from 1. Zero is pure. Compare candidate splits using the children’s impurities weighted by their group sizes.
Worked example: mixed versus pure
Why squares? A 50/50 group has $1-(0.5^2+0.5^2)=1-0.5=0.5$; an all-one-class group has $1-1^2=0$. Concentration lowers impurity.
| Group | Gini impurity |
|---|---|
| Half each class | $0.5$ |
| All one class | $0$ |
A small tree shows its full reasoning as a list of questions. It handles numerical and categorical features, with encoding depending on the implementation. A very deep tree can isolate individual training points and memorize noise; limiting depth or requiring more examples per leaf controls that flexibility.
Forests and boosting: many trees beat one
One tree can fit noise; an ensemble combines trees in two different ways.
| When it wins | Strong tabular baselines: forest for averaging, boosting for successive improvement. |
|---|---|
| When it loses | Less transparent than one small tree; boosting needs validation and tuning. |
A random forest grows trees on randomized samples, considering random subsets of features for splits. Hundreds of trees can vote on a class or average numerical predictions. Randomization reduces how closely their mistakes agree, allowing the aggregate to be steadier than one tree.
Gradient boosting grows trees sequentially. Each adds a correction to the current prediction, guided by the loss. With squared-error regression that means fitting residuals. XGBoost and LightGBM belong to this family, a useful default to compare on tabular problems.
| Forest | Boosting |
|---|---|
| Trees can be built independently | Each tree depends on previous predictions |
| Average predictions or vote | Add loss-guided corrections |
A crowd helps when its errors differ. If every tree makes the same mistake on the same records, voting repeats the mistake. Forest trees train without any other tree’s predictions. A boosting tree needs them: its job is whatever the current ensemble still gets wrong.
| Forest growth | Boosting growth |
|---|---|
| Stabilizes an average | Extends a correction sequence |
Neither arrangement guarantees that errors cancel or that the model generalizes. Use held-out data to judge the result. Feature importance helps inspect a forest but does not explain each prediction as directly as one small tree’s path.
Support vector machines: the widest street
A support vector machine (SVM) asks which separating boundary leaves the widest empty street between two classes. More breathing room makes the boundary less sensitive to small shifts.
| When it wins | Medium-sized classification with a useful margin, including many text features. |
|---|---|
| When it loses | Kernel fitting at large scale; less readable decisions than a small tree. |
The margin is the full gap between two parallel lines touching the nearest points on either side. These touching points are support vectors. For the separable case, other points can move without changing the solution while they remain outside the margin.
Here $\mathbf{w}$ is the weight vector, $\mathbf{x}$ an input, and $b$ the bias; choose their shared scale so the touching points score $-1$ and $+1$.
| Boundary score | Margin-edge scores |
|---|---|
| $\mathbf{w}\cdot\mathbf{x}+b=0$ | $-1$ and $+1$ after normalization |
In words: street width is 2 divided by the length of weight vector $\mathbf{w}$, under the stated score normalization; minimizing that length must still satisfy the separation constraints.
Normalization matters because multiplying every weight and the bias by the same positive number leaves the separating line unchanged. Without fixing the score scale, we could change the formula’s denominator without moving the boundary at all. Requiring the two touching edges to score minus one and plus one ties weight length to a real geometric width.
| Fixed edge scores | Wider street |
|---|---|
| $-1,+1$ | Smaller weight-vector length |
A kernel computes dot products in an implicit feature space where a curved pattern may admit a straight separator. That is a second route to curved boundaries, alongside stacking hidden neurons, and it never builds the lifted coordinates explicitly.
Kernel SVM training gets expensive with many examples, while a linear SVM scales better. Overlapping classes require allowing some margin violations, which balances a wider street against errors.
k-means: finding groups no one labeled
Without labels, k-means clustering partitions points into $k$ groups. Choose the number and starting centers, then alternate assignment and averaging.
| When it wins | Quick grouping, customer segmentation, color compression. |
|---|---|
| When it loses | You must choose k; starts matter; elongated or nested clusters fit poorly. |
Its objective is total within-cluster squared distance:
$$L = \sum \lVert \mathbf{x} - \mathbf{c} \rVert^2$$In words: for each point $\mathbf{x}$, square its distance to assigned center $\mathbf{c}$, then sum over points.
Worked example: one round of assign and update
Use six points: $A(1,1)$, $B(1,2)$, $C(2,1)$ and $D(5,5)$, $E(6,5)$, $F(5,6)$. Start two centers at $\mathbf{c}_1=(0,0)$ and $\mathbf{c}_2=(5,0)$. These low starting centers make the first movement visible; they are guesses, not the final group averages.
A, B, C choose center 1; D, E, F choose center 2. For D, distances are $\sqrt{0^2+5^2}=5$ to center 2 and $\sqrt{5^2+5^2}\approx7.07$ to center 1.
- Assign: choose the nearer current center.
- Update: replace each center with its assigned points’ average.
Average the assigned points coordinate by coordinate:
$$\mathbf{c}_1 \to \left(\tfrac{1+1+2}{3}, \tfrac{1+2+1}{3}\right) = (1.33,\, 1.33) \qquad \mathbf{c}_2 \to \left(\tfrac{5+6+5}{3}, \tfrac{5+5+6}{3}\right) = (5.33,\, 5.33)$$In words: the three near-origin points average to $(1.33,1.33)$; the three distant points to $(5.33,5.33)$, rounded.
Choose $k$ yourself and try several starts; different starting centers can settle into different groupings. Lower loss is a geometric score, not proof that the groups have useful meaning.
PCA: fewer coordinates, most of the spread
Principal component analysis (PCA) describes data with fewer coordinates. Picture a long, thin ellipse of points: most spread lies along its length, little across its width.
| When it wins | Correlated features, compression, 2D visualization, useful distance preprocessing. |
|---|---|
| When it loses | Curved structure or useful signals lying in low-variance directions. |
A principal component is a direction of greatest spread; the next is perpendicular and captures the greatest remaining spread. Keep the long-axis shadow and drop the short-axis coordinate: 2D becomes 1D while preserving most variation in this picture.
Centering makes a coordinate describe displacement from the data’s average, so the chosen axes pass through the cloud instead of an arbitrary origin. Unit-length component directions keep the dot product measuring a coordinate and not an extra rescaling. Both conditions are part of the projection recipe.
- Center: subtract the average point.
- Find axes: rank perpendicular directions by spread.
- Project: keep coordinates on selected unit directions.
First subtract the data’s average point to center the cloud; call a centered point $\mathbf{x}$. Let $\mathbf{u}$ be a unit-length component direction. Its coordinate uses the dot product as a shadow:
$$\text{coordinate of } \mathbf{x} \text{ along component } \mathbf{u} = \mathbf{x} \cdot \mathbf{u}$$In words: dot the centered point with the unit direction to get its signed coordinate along that axis.
The axes come from covariance, which measures how features vary together; the eigenvector calculation is beyond this chapter. PCA can compress correlated columns before another model, or make a 2D view. Nonlinear methods such as t-SNE and UMAP handle different geometry.
| Retained | Not guaranteed |
|---|---|
| Large variance | Predictive usefulness |
Spread and usefulness for a later prediction are different things. A small-variance direction might carry the only distinction between two labels, and PCA never sees it. Keeping most variation can therefore lose an important signal; test the reduced representation on the task. Feature units matter too, because changing their scales changes which directions have the most spread.
Where you'll meet this
| Where | What it looks like | Chapter link |
|---|---|---|
| Labeled tables | Tree ensembles are strong defaults; compare them with a small, inspectable baseline | Choosing your tool |
| Unlabeled tables | k-means chooses groups for a supplied k; PCA keeps high-variance coordinates | Learning signals |
| Images | Local grid structure suggests sharing detectors across space | Seeing machines |
| Sequences and graphs | Word order and contact links carry information that specialized wiring can preserve | Sequences; graphs |
What you now know
- kNN votes among nearby stored labels; its distance rule and neighborhood size determine who votes.
- A decision tree partitions space with feature questions; small trees are readable; very deep trees can memorize.
- Forests average randomized trees; boosting adds successive corrections.
- SVMs favor wide margins, and kernels permit nonlinear boundaries.
- k-means alternates nearest-center assignment and averaging for a chosen number of clusters; different starts may yield different groupings.
- PCA keeps projections along directions of greatest variation; those axes need not retain every task-relevant signal.
Where we’re headed. A pixel’s neighbors, a word’s order, and a node’s links can carry the signal. Specialized networks make those relationships part of their wiring. Next, seeing machines matches a network to an image grid; the toolbox remains our baseline for deciding whether extra complexity earns its cost.