What 'Decoder-Only' Actually Means
At the heart of models like GPT is a 'decoder-only' transformer architecture. To understand what that means, it helps to know what it's missing: an encoder. Original transformer models, often used for translation, had two parts. The encoder would read
and 'understand' a whole sentence in one language, creating a rich summary of its meaning. The decoder would then take that summary and generate a sentence in another language. Decoder-only models threw out the first half. They are built for one core task: given a sequence of text, predict the most likely next piece of text. Think of it less like a translator and more like an incredibly sophisticated autocomplete, one that has read a significant portion of the internet. Its entire existence is dedicated to continuing a pattern.
The Magic of 'In-Context' Learning
One of the most startling behaviors for new practitioners is 'in-context learning'. You can give the model a few examples of a task in the prompt—like 'Translate English to French: sea otter -> loutre de mer'—and it can then perform the task on a new example you provide, all without any formal retraining. This feels like rapid, on-the-fly learning, but it's a direct result of the decoder-only design. The model isn’t learning a new skill; it’s recognizing the pattern in your prompt and continuing it. It sees the examples and your new query as one long piece of text and simply calculates the most probable continuation, which happens to be the correct answer in the format you demonstrated. No weights are updated; it's all just a massively scaled-up game of pattern completion.
Emergent Abilities: From Coherence to Chaos
A model trained only to predict the next word shouldn't be able to write a coherent essay, summarize a document, or answer a complex question—yet they do. These are known as 'emergent abilities', capabilities that aren't explicitly programmed but appear as the models get bigger and are trained on more data. The long-form coherence is surprising because each word is just a statistical guess. But by learning the statistical patterns of trillions of sentences, the model implicitly learns grammar, style, and even how to structure arguments. The flip side of this is the model's tendency to 'hallucinate' or invent facts. Because its goal is to generate plausible text, not to state truths, it will confidently produce incorrect information if that information fits the pattern it's trying to complete.
Why Simplicity Won the AI Race
For a time, it wasn't obvious that the simpler decoder-only architecture would become dominant. But its simplicity turned out to be its greatest strength. By unifying every task into a 'next-token prediction' problem, training became more streamlined and scalable. You no longer needed carefully paired datasets (like sentence-to-translation pairs); you could just feed the model vast quantities of raw text. This allowed for unprecedented scale, and at that scale, decoder-only models started matching or exceeding more complex architectures even on tasks like summarization. They are not as good at understanding tasks as encoder-based models, but their generative power and adaptability have made them the standard for most modern large language models.
