Seeing Machines

Why a picture breaks a plain network

A photo, to a computer, is nothing more than a grid of numbers: each pixel a brightness value (one number for gray, three for color). So here is the obvious idea — pour those numbers into the plain stack of neurons we built two chapters back in the neuron and ask it "cat or dog?" Try exactly that, and two disasters happen at once. Fixing both of them is the whole idea of this chapter.

The first disaster: the numbers explode. To feed a plain network you must first flatten the grid — unroll the whole image into one long list of numbers. A modest $100 \times 100$ grayscale image is already $10{,}000$ numbers. Wire that to a first hidden layer of just $10{,}000$ neurons and you need $10{,}000 \times 10{,}000 = 100$ million weights in the first layer alone — that is the matrix-times-vector count from the math toolkit. Every one of those weights is a dial the network must learn. A real photograph would demand billions of dials before the second layer even began. It does not fit — not in memory, not in a lifetime of training.

The second disaster: flattening shreds the picture. Two pixels sitting side by side in the image — plainly part of the same whisker — can land far apart in the flattened list, while pixels from opposite corners of the photo end up as neighbors. The network is handed $10{,}000$ apparently unrelated numbers and told "good luck." The single most important fact about an image, that nearby pixels belong together, has been thrown in the bin before learning even starts.

Both disasters have a single cure. Instead of one giant layer that stares at the whole flattened image at once, use a tiny detector — just a handful of weights — that looks at one small patch at a time and slides across the whole picture, reusing the same weights everywhere. That sliding detector is a convolution, and a network built from them is a convolutional neural network, or CNN. By the end of this chapter you will have slid one across an image with your own hands.

Sliding a filter across the picture

Let us build the machine one patch at a time, with numbers small enough to check by hand. Our tiny image is a $5 \times 5$ grid holding a single clean vertical edge. Every row is identical — bright on the left, dark on the right — written $[1, 1, 1, 0, 0]$, where $1$ is bright and $0$ is dark. That is the whole picture, and it contains exactly one feature: a vertical edge, where the brightness drops between the third and fourth columns.

Now the detector. A filter (also called a kernel) is a small grid of weights; ours is $3 \times 3$. To detect a vertical edge, use

In words: this filter adds up the left column of whatever patch it covers and subtracts the right column. It gives a big number where the left is bright and the right is dark — an edge — and zero where both sides match. A filter is nothing exotic: it is a neuron from the neuron with only nine weights and a small field of view.

Run it at one position, every digit on the table. Lay the filter over the patch that straddles the edge — the three rows are all the same, $[1, 1, 0]$ — and multiply matching cells, then add. This is exactly the dot product, just laid out in a square instead of a line:

In words: line the filter up over a patch, multiply each weight by the pixel under it, and add it all up — one number that says how strongly this patch matches the pattern.

Now slide and repeat. Move the filter one step right and compute again; step down; cover every position where the filter fits fully inside the image. Each position produces one number, and those numbers assemble into a new, smaller grid called a feature map. On our image the full feature map is

In words: a $3$ wherever the filter sat over the edge, a $0$ over the flat bright region. To prove the zero, do one cell in the open — over the flat-bright patch (the three leftmost columns, all $1$s), each row gives $(1)(1)+(1)(0)+(1)(-1) = 0$, so the sum is $0$. The feature map is bright exactly where the image changes and dark everywhere else: the filter has turned "raw pixels" into "where the vertical edges are."

Two small, honest details, named and moved past. First, the feature map is smaller than the image because the filter cannot hang off the edges; real CNNs often pad the border with zeros to keep the size, a piece of bookkeeping, not a concept. Second, after the sliding sum, each feature-map number is passed through an activation — usually the ReLU from drawing boundaries, which keeps positives and zeros out anything negative — so the map holds "how present is the feature, never below zero." Neither detail changes the picture: slide, dot-product, squash.

A filter is a feature detector

A filter is specific: it detects one pattern and ignores the rest. To prove it, take the very same image and slide a horizontal-edge filter across it —

In words: this filter takes the top row of a patch and subtracts the bottom row. Because our image has no horizontal edges — every row is identical — the top and bottom always match, so every position gives $0$. The feature map is all zeros.

Now the leap that powers the whole field: the filter's nine weights are the pattern it detects — and, crucially, those weights are not hand-designed, they are learned. We chose $[[1,0,-1],\dots]$ by hand to make a clean example, but in a real CNN the filter weights start random and are trained by gradient descent (from teaching the network) until the network discovers, on its own, which little patterns are worth detecting. Nobody tells it "look for vertical edges"; the training data does.

One filter finds one pattern, so a convolutional layer uses many filters side by side — say $64$ of them — each sliding over the same image and each producing its own feature map. One learns vertical edges, another horizontal, another a patch of a particular color, another a small curve. The layer's output is a stack of $64$ feature maps: the same image re-described as "here are all the little patterns present, and where." That is the honest meaning of "the network learns features."

The trick: share the weights

Return to the first disaster and cash in the savings, with hand-checkable numbers. Suppose we want to turn our $5 \times 5$ image ($25$ pixels) into a $3 \times 3$ feature map ($9$ outputs). A plain fully connected layer would wire every one of the $25$ inputs to every one of the $9$ outputs:

In words: the dense layer needs a separate weight for every input-output pair — $225$ dials. A convolution produces the same nine outputs with a single $3 \times 3$ filter — nine weights, reused at all nine positions. And the gap explodes with image size: on a real megapixel photo the dense layer needs billions of weights while the filter still needs nine.

That reuse has a name: weight sharing. And weight sharing is not merely cheaper — it is right. A vertical edge is a vertical edge whether it sits in the top-left corner or the bottom-right. It would be absurd to learn "detect an edge here" separately from "detect an edge there," so we do not: one filter, learned once, is slid everywhere. This means the network detects a pattern no matter where it appears — a built-in assumption called translation equivariance, and it is exactly the right prior belief for pictures.

You have met this shape of idea before, and you will meet it again. Here in seeing machines we share weights across space; the next chapter will share weights across time. Sharing a small set of weights across all positions is the single trick that lets a network exploit the structure of its data instead of drowning in a flat list of numbers. Same trick, different axis.

Pooling, and building a sense of sight

After a convolution, feature maps are still large, and we rarely need pixel-perfect location — knowing there is an edge "around here" is enough. Pooling shrinks a map by summarizing each small block into a single number. The common form, max pooling, keeps only the strongest response in each block:

In words: chop the map into little $2 \times 2$ blocks and keep the loudest number in each — the top-left block keeps its max, $3$; the bottom-right block keeps $4$ — half the size, the strong signals preserved.

Pooling buys two things. It shrinks the data — fewer numbers for later layers, less computation — and it adds a little position-tolerance: if the edge shifts by one pixel, the block's maximum usually does not change, so the network becomes slightly blind to exact location. For recognizing what is in an image, that blindness is a feature, not a bug.

Now stack it all into the shape of a real CNN. A CNN alternates convolution (find patterns), activation (ReLU), and pooling (shrink), over and over — and the magic is what the stacking does. The first layer's filters find edges. The second layer's filters slide over the edge-maps and find combinations of edges — corners, curves, textures. The third finds combinations of those — an eye, a wheel, a letter. The last finds whole objects. Each layer composes the previous layer's features into richer ones — precisely the "depth builds complex shapes cheaply" argument from the neuron, now made visual.

Name that growing view once. A first-layer neuron sees a $3 \times 3$ patch; a neuron two layers up sees a patch of patches, so it effectively looks at a much larger region of the original image — its receptive field grows with depth. Shallow neurons see tiny details; deep neurons see big structure. That is how a pile of $3 \times 3$ filters ends up understanding a whole face.

See it move

The video runs the whole argument in one motion: watch the $3 \times 3$ vertical-edge filter slide across the $5 \times 5$ image, computing a dot product at each stop and lighting up the feature map — $0$ over the flat region, $3$ over the edge, the same numbers you just did by hand. Then watch a second, third, and fourth filter build a stack of feature maps, and see how stacking layers turns edges into objects.

Now you drive it. The playground below lets you be the filter. A small image sits on the left, an editable $3 \times 3$ filter in the middle, and its live feature map on the right. Move the filter's window over the image and watch the nine multiply-adds and their sum; edit the filter's weights and watch the feature map redraw instantly.

Where you'll meet this

Zoom out to where CNNs actually live. For roughly a decade, convolutional networks were how machines saw: reading handwritten digits on cheques, spotting tumours in scans, letting a car's camera find pedestrians, sorting billions of photos. Any time the data is a grid where nearby cells belong together — an image, a spectrogram of sound, a heat-map — convolution is the natural first tool, for exactly the two reasons this chapter built: it shares weights so it fits, and it respects locality so it learns the right thing.

Now the honest present-day thread. The idea of attention that powers language models (see the sister site's chapter on attention) has spread into vision too — "vision transformers" now rival or beat CNNs on the biggest datasets by treating an image as a sequence of patches. But the convolution's core insight did not die; it got absorbed. The lesson that a network should share weights and exploit local structure is permanent, whatever the winning architecture is called this year.

And that sets up the next two chapters precisely. An image is a grid, and we shared weights across space. But a sentence is a sequence, where order carries meaning and the input has no fixed length — "the dog chased the cat" and "the cat chased the dog" are the same words in a different order and opposite meanings, and a grid filter does not fit. And a network of contacts is a graph, where each node has a different number of neighbors and there is no grid at all. The next chapter shares weights across time to read sequences; the one after generalizes the whole idea to arbitrary graphs. Matching the wiring to the shape of the data is the thread running through all of Part III.

Close warm. You have slid a filter across an image, computed a feature map by hand ($0$ on the flat, $3$ on the edge), counted the $225$-to-$9$ weight saving that makes vision affordable, and seen how stacking edges into objects lets a pile of nine-weight detectors recognize a face. That is how a machine learns to see — not magic, just a dot product, slid everywhere, stacked deep.