Autoencoders

AnAn autoencoder is trained to do something that sounds pointless stated plainly: reproduce its own input as its output. The point isn't the reconstruction itself — it's what the network is forced to learn in order to do that reconstruction well.

The Bottleneck Is the Whole Point FoundationalKnowledge that endures for decades — core principles

AnAn encoder compresses the input down to a small latent representation — far fewer numbers than the input had — and a decoder tries to reconstruct the original from just that small representation. If the latent layer were the same size as the input, the network could trivially learn to copy every value straight through, unchanged, and "reconstruction loss" would hit zero without the network having learned anything useful about the data's actual structure. The bottleneck — a latent space deliberately much smaller than the input — rules that shortcut out. To reconstruct a compressed image well, the network has to discover and retain whatever structure in typical inputs makes compression possible at all: that neighbouring pixels tend to be similar, that certain shapes recur, that most of the input's variation sits in far fewer independent directions than it has raw pixels.

Dimensionality Reduction, Denoising, Sparse Codes FoundationalKnowledge that endures for decades — core principles

TrainedTrained successfully, the latent representation itself becomes useful independent of the decoder — a compact, learned summary of each input, usable for visualisation, for comparison between examples, or as a smaller input to some other model entirely, the same dimensionality-reduction job PCA does with a purely linear method rather than a learned nonlinear one. A denoising autoencoder is trained on a deliberately corrupted input but asked to reconstruct the clean original, which forces the latent representation to capture genuine structure rather than passively encoding whatever noise happens to be present. A sparse autoencoder keeps a latent space as large as the input but adds a penalty encouraging most latent units to sit at zero for any given input, so only a small active subset does the actual representing each time — a different route to the same goal of ruling out the trivial copy-through solution.

Ordinary Autoencoders Cannot Generate; Variational Ones Can FoundationalKnowledge that endures for decades — core principles

AnAn ordinary autoencoder's latent space has no reason to be smooth or well-organised — nothing in its training objective stops two very similar inputs from encoding to two very different, unrelated points, which means a random point in that latent space, decoded, is as likely to produce nonsense as something recognisable. A variational autoencoder changes what the encoder outputs (a small distribution around each point, rather than one fixed point) and adds a second training term pulling every one of those distributions toward a shared, simple shape, which makes the whole latent space smooth and samplable — the specific mechanism behind this section's companion page on Generative Models, covered there in full alongside GANs and diffusion models.

Related Topics

• Generative Models: VAEs, GANs, and Diffusion — the variational autoencoder, and two further generative families, covered in full.
• Unsupervised Learning: Clustering and Dimensionality Reduction — PCA, the linear counterpart to this page's learned, nonlinear compression.
• Self-Organising Maps — another unsupervised method that builds its own low-dimensional representation of the input, by a completely different (competitive, prototype-based) mechanism.