Part III · Ch. 19 — The Autoencoder

Part III · Chapter 19 of 29

The Autoencoder

Compress a digit to two numbers and reconstruct it


What shape does it eat?

What if the answer we ask a model for is its own input, forced through a narrow passage first? Copying every entry straight through would be easy. Rebuilding it from fewer numbers makes the model choose what to keep. That choice is inspectable with the same matrix arithmetic and training records we used for classification.

An autoencoder is trained to reconstruct its input. The encoder maps an input into a compact representation, a set of numbers used by the next computation. The latent is the representation handed to the decoder, which maps it back to the output shape. A bottleneck restricts that representation’s width or information capacity. Reconstruction loss scores the difference between the original input and the reconstruction.

Top: the hand example’s 4→2→4 encoder and decoder matrices with its input and reconstruction. Bottom: the digit autoencoder’s layer shapes, 64→16→2→16→64, with each activation named.
An encoder and decoder are trained together to preserve input information.

The digit autoencoder we trained takes flattened 8×8 digit pixels, so one input has 64 entries. Its encoder maps 64→16 with tanh, then 16→2 with a linear layer. Its decoder maps 2→16 with tanh, then 16→64 with sigmoid pixel outputs. The target is the original pixel vector. Digit labels color the displayed map and never enter the reconstruction objective. The task is self-supervised, because the data supplies its own target.

The target is the input

The target matches the input, so a useful solution has to keep information that helps rebuild many examples. The decoder learns how to turn nearby latent coordinates into output vectors. That doesn’t guarantee that distances in the latent plane match distances between original images, or that one direction has a convenient human meaning. Those are properties to inspect, not assume.

The labeled unit lattice from chapter 2, drawn in the hand latent plane, where each point decodes as (h1,h1,h2,h2). The point (2,3) is highlighted with the two different hand inputs that both encode there.
Two different inputs can share the same compressed representation.

The hand lattice reuses the coordinate identities from space that stretches, so each point can be decoded instead of read as an abstract scatterplot. With our hand decoder, latent (h₁,h₂) becomes (h₁,h₁,h₂,h₂). Moving horizontally changes the first pair of output entries; moving vertically changes the second pair. The trained decoder is more complicated, but it too turns each latent point into one definite output vector.

Encode, decode and measure what was lost

Use input x=(1,3,2,4). Our hand encoder averages adjacent pairs with rows [1/2,1/2,0,0] and [0,0,1/2,1/2]. The bias is zero and there is no bend. The decoder rows are [1,0],[1,0],[0,1],[0,1]. The whole toy is linear, which makes the discarded information easy to name.

Encode, decode, and score

The latent is ((1+3)/2,(2+4)/2)=(2,3). The decoder duplicates each latent coordinate, giving (2,2,3,3). Subtracting the original gives errors (1,−1,1,−1). Squaring removes their signs and gives (1,1,1,1). The mean reconstruction loss is (1+1+1+1)/4=1. All four errors count equally, because this hand objective averages over all four entries.

A collision in the representation

Now encode (2,2,3,3). Its pair averages are again (2,3), and its reconstruction is (2,2,3,3) with loss zero. Two different inputs reached the same latent. After that collision the decoder can’t know which original we meant. The encoder removed the within-pair differences, so no later deterministic calculation recovers them for both inputs from that one latent vector.

Bold x names the original vector, bold h the latent, and x with a hat the reconstruction. The count n is the number of input entries averaged over. The encoder and decoder stand for the real sequences of operations, activations included:

$$\mathbf{h}=\mathrm{encoder}(\mathbf{x}),\quad\hat{\mathbf{x}}=\mathrm{decoder}(\mathbf{h}),\quad L=\frac{1}{n}\sum_i(\hat x_i-x_i)^2$$

In words: compress the input, reconstruct it, then average its squared pixel errors.

Two more hand inputs

The video excerpt later in this chapter uses the same averaging and duplication matrices with two other inputs. For (6,6,2,2), the latent is ((6+6)/2,(2+2)/2)=(6,2), and decoding restores the input unchanged. For (6,5,2,3), the latent is (11/2,5/2)=(5.5,2.5), the reconstruction is (5.5,5.5,2.5,2.5), and every squared error is 0.25. Their mean is 0.25.

Replay the learned representation

The digit autoencoder learns both mappings together from the pixel loss. Class labels play no part in that optimization, so nothing instructs the model to separate digit classes in the latent plane. A cluster can still appear, because related images share reconstructible structure. Whether it does shows up in the saved coordinates and reconstructions; the picture alone doesn’t name the axes.

Every digit’s two latent coordinates at the start and at the end of training, on fixed axes, colored by digit label, with a strip of saved probe digits and their reconstructions beneath.
Latent coordinates and reconstructions saved while the digit autoencoder trained.

At step 3000, the digit autoencoder has training loss 0.0383 and held-out reconstruction loss 0.0409. These are mean squared pixel reconstruction errors. They are not classification accuracy, and multiplying one by a hundred doesn’t make it a percentage of correctly recognized digits.

An excerpt from the diffusion book’s compression video, working the same averaging and duplication by hand.

Where this shows up when you train

A reconstruction can look broadly recognizable while losing contrast at particular pixels. Compare differences at the same locations rather than trusting a general impression of similarity. The figure puts the saved probe originals beside their final reconstructions and marks the probe with the largest mean error. Only the probes were saved, so it is the worst probe, not necessarily the worst image in the dataset.

Probe digits beside their final reconstructions and grids of the pixel differences, with the probe that has the largest reconstruction error marked as the worst of the probes.
Reconstruction error identifies what this model failed to preserve.

Each probe is the first training example of its digit label. Another image with the same digit label is a different probe. The residual grid subtracts those original values from the saved reconstruction values, so positive and negative errors are both visible. Compare a bright residual cell with the matching original and reconstructed pixel before you describe lost detail.

For a new dataset, choose preprocessing, a reconstruction objective, and an independent evaluation set before you judge the latent picture. Keep the original scale so a pixel difference means something. If the eventual task is classification or anomaly review, evaluate that task directly too. The compact plane is a good place to inspect behavior.

What you now know

  • An encoder compresses and a decoder reconstructs.
  • A bottleneck can map different inputs to the same code.
  • Reconstruction quality and label separation are different properties.

Where we’re headed

The model recipes are visible now. Next: turning real files into the shapes they need, without leaking evaluation information. Continue the story.