Last updated: 2026-10-09

U
Undergraduate level

Empiricism, Rationalism, and Machine Learning

Train a model on nothing but examples, with no hand-written rules, and it looks like the empiricist's dream made literal: a mind built entirely from experience, Locke's blank slate finally realised in silicon rather than only argued for in prose. This page takes that appearance seriously enough to show exactly where it breaks down — not because the empiricist story is wrong about what the training data contributes, but because no learner, biological or artificial, actually starts with nothing.

The Empiricist Reading, and What It Gets Right FoundationalKnowledge that endures for decades — core principles

Locke's empiricism holds that all ideas ultimately derive from experience — sensation and reflection — with no innate content supplied in advance1. A trained model's weights genuinely are shaped entirely by exposure to data, with no explicit rule ever hand-written into them, which is real and worth taking seriously rather than dismissing as a superficial resemblance: the specific content a model ends up representing — which concepts cluster together, which continuations are likely — comes from the training distribution, not from the architecture's designer having anticipated and encoded it.

What Was Already Decided Before Any Data Arrived FoundationalKnowledge that endures for decades — core principles

None of that content-from-data story explains how the learning happens at all, and everything in that second question is settled before a single training example is seen:

  • Architecture fixes what kinds of functions the system can represent at all — a convolutional structure can represent translation-invariant features cheaply and other structures only expensively or not at all, regardless of what the data contains.
  • Objective function fixes what counts as a good answer during training, which silently rules out optimising for anything the objective doesn't reward, however present it is in the data.
  • Optimisation procedure fixes which of the many configurations that would fit the training data the search actually finds, since most learning problems have far more solutions consistent with the data than the one training converges on.
  • Tokenisation and data selection fix what counts as a unit of input and which parts of the world are even sampled, before any statistical learning begins over the result.
  • Initial conditions and representational constraints fix the starting point and the shape of the space being searched, which shapes which patterns are easy to find and which are effectively invisible to the search.

Wolpert and Macready's no-free-lunch theorem gives this an exact, formal edge rather than leaving it as a qualitative observation: averaged across every possible problem a learning algorithm could face, no algorithm outperforms any other, including random guessing2. Any algorithm's actual, practical superiority on the problems it's actually used for is entirely a function of its built-in assumptions matching the structure of those problems — there is no assumption-free learner that simply "reads off" patterns from data in general, because without some prior restriction on which patterns are even eligible to be found, every possible pattern (including pure noise) is as findable as every other.averaged across all possible problems, no learner beats any other

A Cautious Kantian Comparison FoundationalKnowledge that endures for decades — core principles

Kant's reply to Locke's empiricism and Hume's scepticism argued that experience alone cannot produce knowledge without the mind already supplying certain organising categories — space, time, causation — that structure any experience before it can be understood at all3. The comparison to a network's architecture is worth making carefully rather than literally: nothing here claims a convolutional filter or an attention mechanism is a Kantian category, or that the comparison explains either side better than it was already understood. What the comparison usefully asks is a narrower, structural question — given that Kant is right that some pre-experiential structure is required for experience to yield knowledge at all, what does the analogous structure look like for a trained model, and does identifying it change what the model's "knowledge" should be taken to mean? The answer this page gives: architecture plays a Kant-shaped role (conditioning what can count as a pattern at all) without filling it with Kant-specific content (there is no evidence networks converge on anything like space, time, or causation as organising categories specifically).

Note well. The claim here is structural, not substantive: a trained system's architecture plays the role of pre-experiential organising structure, the way Kant argued some structure must exist. This does not mean the specific structure a network has resembles Kant's specific categories, and no further claim about neuroscience or Kant's actual psychology should be read into the comparison.

Strongest Objection: This Makes the Distinction Too Easy to Win FoundationalKnowledge that endures for decades — core principles

A fair complaint: "no learner is structure-free" is close to unfalsifiable, since any learner that works at all has some structure by definition, which risks making the empiricist position a straw target nobody actually held in its strongest form. The more defensible empiricist position was never "zero structure" but "minimal, generic structure, with almost all specific content coming from experience" — and the genuine, substantive question this page's architecture list raises is whether modern deep networks' structure is minimal and generic in that sense, or whether it smuggles in more specific, consequential assumptions than the empiricist framing admits. That's an empirical question about particular architectures, not one this page settles by pointing out that some structure exists.

Provisional Conclusion FoundationalKnowledge that endures for decades — core principles

Machine learning is neither pure empiricism nor pure rationalism. Data genuinely constrains what a trained system ends up representing, the way the empiricist story says; architecture, objective, optimisation, and representational choices genuinely constrain what can count as a pattern, an error, or a successful generalisation in the first place, the way the rationalist insistence on prior structure says. Asking which side a given system belongs to is less useful than asking, for that specific system, exactly which choices were made before training started, and what those choices ruled out before any data had a chance to matter.

Questions for Further Thought

  • Is there a meaningful sense in which one architecture's built-in assumptions are more "minimal and generic" than another's, or does the no-free-lunch result make that comparison incoherent?
  • If a system's architecture plays a Kant-shaped structural role, does that licence any further comparison to Kant's specific claims about space, time, and causation — or does the analogy stop exactly where this page says it does?
  • Which of this page's five pre-training choices (architecture, objective, optimisation, tokenisation/data, initial conditions) seems most responsible for a specific capability or blind spot you've actually observed in a deployed system?

Further Reading

  • Locke, J. (1689). An Essay Concerning Human Understanding.
  • Kant, I. (1781). Critique of Pure Reason.
  • Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82.

References

  1. Locke, J. (1689). An Essay Concerning Human Understanding. ↩

  2. Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82. ↩

  3. Kant, I. (1781). Critique of Pure Reason. ↩