Understanding Large Language Models
Underneath every capability an LLM appears to have — answering questions, writing code, holding a conversation — it is doing one specific and much narrower thing on every single step: predicting which token (roughly, a word-fragment) is statistically most likely to come next, given everything written so far. Everything else — the appearance of reasoning, the appearance of knowledge, the appearance of a personality — is a consequence of that one mechanism operating at enormous scale, not a separate capability bolted on top of it. Holding this firmly in mind is the single most useful thing a non-programmer can know about LLMs, because it explains both why they are remarkably capable and exactly where and why they go wrong.
What an LLM Actually Does
An LLM is built and trained to do next-token prediction over enormous quantities of text — a large fraction of the public web, books, and other written material — so that, given a sequence of tokens, it can estimate how likely each possible next token is. Generating a full response is just this single-step prediction repeated: predict the most likely next token, add it to the sequence, predict the next one after that, and so on, until the response is complete. There is no separate "understanding" module consulted at any point in this process, and no built-in mechanism that checks the output against reality before it is produced — the model is being fluent, not necessarily being right, and those two properties are correlated but not the same thing.
Attention: Reading Everything at Once
The specific architecture that made today's LLMs possible is the Transformer, introduced in a 2017 paper whose title stated its central claim outright: "Attention Is All You Need" [1]. Its key idea, put in plain terms, is a mechanism called self-attention: when the model processes any given word in a passage, it does not read strictly left-to-right the way a human does on a first pass — it simultaneously weighs every other word in the available context against that word, learning which of them matter most for interpreting it. This is what lets a model correctly resolve, in "the trophy didn't fit in the suitcase because it was too big," that "it" refers to the trophy rather than the suitcase — a judgement that depends on weighing several words at once, not reading in strict sequence. Because this weighing can be computed for every word in a passage in parallel, rather than one at a time in sequence, the Transformer architecture also turned out to scale far more efficiently to enormous training runs than the older architectures it replaced — which is a large part of why the current generation of very large, very capable models became possible at all.
generated so far"] --> B["Self-attention:
weigh every token against
every other token"] B --> C["Predict the single
most likely next token"] C --> D["Append it, repeat"] D --> A
Two Phases of Training: Pretraining and Alignment
A model trained purely to predict the next token over raw internet text is not, by default, a helpful assistant — it is a very good imitator of whatever kind of text preceded a given passage in its training data, which might be a Wikipedia article, an angry forum post, or a piece of misinformation, all imitated with equal fluency. The models sold today as chat assistants go through a second phase after this initial "pretraining": instruction-tuning, where the model is further trained on examples of the kind of helpful, well-formed responses it should produce, often combined with reinforcement learning from human feedback (RLHF) — human raters compare candidate responses, and the model is adjusted to make highly-rated response styles more likely [2]. This second phase is why a modern chat assistant sounds cooperative and structured rather than like a random slice of the internet — but it is worth being precise about what it changes: RLHF shapes the style and helpfulness of responses far more directly and reliably than it guarantees their factual accuracy, because human raters are judging how a response reads, not independently re-verifying every claim in it.
Emergent Capabilities and Their Limits
As LLMs have grown larger, they have picked up capabilities — multi-step arithmetic, translation between languages barely present in training data, following complex multi-part instructions — that smaller versions of the same architecture simply could not do at all, not just did poorly; performance on some tasks looks close to random until a model crosses a certain scale, then rises sharply [3]. This phenomenon, termed "emergent abilities," is genuinely useful to know about, but it is also actively debated rather than settled: a widely cited 2023 analysis argued that at least some apparent emergence is a measurement artefact — a side-effect of using metrics (like exact-match accuracy) that are themselves discontinuous, and that smoother metrics on the very same models and tasks often show gradual, predictable improvement rather than a sudden jump [4]. The practical lesson for evaluating any specific claim about "emergent" AI capability is to ask what exactly was measured and how, before accepting that a capability appeared "suddenly" rather than gradually.
What LLMs Get Wrong
Three limitations matter most for using LLMs critically and are worth naming precisely rather than gesturing at vaguely. First, hallucination: LLMs regularly generate fluent, confident, entirely fabricated content — invented citations, invented facts, invented API calls — because nothing in the next-token-prediction mechanism distinguishes "plausible continuation" from "true statement"; a comprehensive survey of the phenomenon across natural-language-generation systems documents this as a structural property of the technology, not an occasional bug fixable by a patch [5]. Second, knowledge cutoffs: a model's knowledge of the world is frozen at whenever its training data was collected, and it generally cannot distinguish "I don't know this because it happened after my training" from "I don't know this at all" unless specifically told the date of its cutoff — it will often guess fluently rather than say so. Third, prompt sensitivity: how a question is phrased — word choice, formatting, even something as superficial as whether a list uses numbers or letters — can measurably change an LLM's answer, sometimes drastically, even when the underlying question is logically identical; this has been directly measured and quantified across multiple models and formatting variations, not just anecdotally observed [6]. None of this means LLM output is useless — it means LLM output needs the same posture a careful reader brings to an unverified secondary source: useful as a starting point, not a substitute for checking.
References
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NeurIPS 2017), 5998–6008.
- Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 27730–27744.
- Wei, J., Tay, Y., Bommasani, R., et al. (2022). Emergent Abilities of Large Language Models. Transactions on Machine Learning Research.
- Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 55565–55581.
- Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730
- Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2023). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. arXiv:2310.11324