Before GPT: The Original Transformer
To understand the revolution, you first have to know what came before. In 2017, a landmark paper from Google researchers titled "Attention Is All You Need" introduced the Transformer architecture. Think of it like a highly efficient translation agency.
It had two main divisions: an 'encoder' and a 'decoder'. The encoder's job was to read an entire sentence (say, in English) and create a rich, comprehensive understanding of its meaning. The decoder would then take that understanding and begin generating the translated sentence (say, in German), token by token, while constantly referring back to the encoder's complete picture. This two-part system was powerful for tasks like translation and summarization, where you have a distinct input that needs to be converted into a distinct output.
The 'Less Is More' Revolution
The game-changing insight behind models like GPT was deceptively simple: What if you just threw away the encoder? Researchers at OpenAI and elsewhere discovered that for many tasks, you didn't need a separate module to 'understand' the input. Instead, you could use a 'decoder-only' architecture. This model does one thing, and it does it obsessively well: it predicts the next word in a sequence. You give it a prompt, and it just keeps guessing the most probable next token based on everything it has seen before. This process is called autoregression. Every time it generates a new word, it adds that word to the input sequence and runs the whole process again. It's always just continuing a thought, never stopping to form a separate 'understanding' like the old encoder-decoder models did.
The Power of Predicting the Next Word
This might sound like a limitation, but it turned out to be the key that unlocked the generative AI boom. This 'decoder-only' approach, with its single-minded focus on text continuation, proved to be incredibly versatile. Suddenly, almost any task could be framed as a completion problem. Want a summary? Give the model an article and the prompt, "Tl;dr:". Need code? Describe the function you want and let it 'complete' the script. This unified approach simplified training immensely. Instead of needing carefully curated pairs of inputs and outputs (like English-to-German sentences), researchers could just feed the model trillions of words from the open internet and train it to predict what comes next. This scalability is a major reason models like GPT, Llama, and Claude became so powerful.
What This Unlocked: Conversation and Creativity
The decoder-only architecture is inherently suited for conversation. Because the model simply continues a sequence, it can handle a flowing, back-and-forth dialogue naturally. This structure also excelled at something called 'in-context learning.' You can give it a few examples of a task within the prompt itself, and it intuits the pattern without needing to be retrained. But perhaps most profoundly, focusing only on next-token prediction led to emergent abilities in creativity and reasoning. Because the model is trained on such a vast and diverse dataset, its predictions are not just grammatical; they are often insightful, creative, and contextually aware in ways that have fundamentally changed our expectations of artificial intelligence. It's a system that, by simply learning to continue patterns, taught itself to reason, write, and code.
