Associative-Memory and Hopfield Networks

AA Hopfield network, introduced by Hopfield in 1982, answers a different question than every other architecture in this section¹: not "what label does this input get," but "given a noisy or partial version of something I've stored, can the network settle back onto the clean original?" It's a bridge between the memory-based framing of RAM networks and the recurrence of the previous page, built from neither addressed memory cells nor a one-way sequence, but a fully-connected web of units that feed back into each other until the whole network stops changing.

Storing Patterns as Attractors FoundationalKnowledge that endures for decades — core principles

EveryEvery unit in a Hopfield network is connected to every other unit (but not to itself), each connection carrying a weight, and every unit's state is simply +1 or −1. "Training" sets those connection weights, in the classic formulation, directly from the patterns to be stored — no gradient descent, no iterative search — so that each stored pattern becomes a stable configuration: a state where, if every unit checks its neighbours and recomputes its own value, nothing changes. These stable configurations are the network's attractors.

Recall by Settling FoundationalKnowledge that endures for decades — core principles

RecognitionRecognition means loading a noisy or incomplete pattern as the network's starting state, then repeatedly updating units (one at a time, or all together) according to the weighted sum of their neighbours' current states. Each update moves the network's overall state a little closer to whichever stored attractor is nearest to the starting point, the same way a ball placed anywhere on an undulating surface rolls downhill into whichever dip is closest — the energy landscape this process is usually described against, where stored patterns sit at local minima and the update rule is guaranteed to decrease (or hold) the network's total energy at every step, never increase it.

Capacity and Spurious States Applied / MethodologicalKnowledge with a 5–10 year half-life — stable practice

AA Hopfield network of \(N\) units cannot store an unlimited number of patterns reliably — roughly \(0.14N\) random patterns is the widely-cited practical capacity limit, after which stored patterns start to interfere with each other and recall becomes unreliable. Overloading a network, or storing patterns that are too similar to each other, can also create spurious states: stable configurations the network settles into that were never deliberately stored at all, artefacts of how the individual stored patterns' attractor basins combine and sometimes merge or create new dips of their own.

Contrast With RAM Networks FoundationalKnowledge that endures for decades — core principles

BothBoth families store something other than weights, and it's worth being precise about how differently they do it. A RAM network uses the input bits directly as an address into explicit memory cells — recall is a single table lookup, no iteration, no settling. A Hopfield network stores nothing at an address at all; the pattern is encoded implicitly in the connection weights between every pair of units, and recall is an iterative dynamical process that takes several steps to settle, with no guarantee of how many.

Related Topics

• Weightless Neural Networks and RAM Networks — the address-based alternative this page's energy-settling approach is directly contrasted against above.
• Recurrent Neural Networks — another architecture built from feedback connections, processing a sequence step by step rather than settling toward a single fixed attractor.

References