An Idea as Old as Cognition
The core concept behind Chain-of-Thought (CoT) prompting is almost comically simple: to solve a hard problem, you break it down into smaller, logical steps. It’s how humans have thought for millennia. If you're solving a math word problem, you don't just
guess the answer; you identify the variables, set up the equation, and solve it step by step. Early artificial intelligence researchers were well aware of this. For decades, they tried to build systems that could mimic this structured, sequential reasoning. They coded explicit rules and logical frameworks, hoping to create a machine that could 'think' like a person. Yet, for all their efforts, these early AI systems remained stubbornly brittle and inflexible. They could follow narrow, pre-programmed paths but couldn't handle the ambiguity and common sense required for true multi-step reasoning. The idea was sound, but the technology simply wasn't there.
The Brick Wall of Small Models
For decades, the dream of an AI that could show its work was stuck. The problem wasn't the concept, but the vessel. Early AI models, and even the language models of just a few years ago, were too small. While they could mimic language patterns, they lacked a deep, nuanced understanding of the world. They could predict the next word in a sentence, but they couldn’t truly comprehend the logical connections between ideas. Asking a smaller model to 'think step by step' was like asking a calculator to write a poem. The calculator has rules for numbers, but it has no concept of metaphor, rhythm, or emotion. Similarly, older models had rules for syntax but lacked the rich, internal world model needed to make logical leaps. They would often get stuck, invent nonsensical steps, or confidently arrive at the wrong answer. The 'thinking' part was missing.
The Real Reason: Emergent Abilities
The game changed not with a new algorithm, but with a new phenomenon: scale. As tech giants built progressively larger language models—with hundreds of billions of parameters trained on continent-sized datasets—something unexpected began to happen. These massive models started displaying 'emergent abilities'. An emergent ability is a skill that isn't explicitly trained into the model and doesn't exist in smaller versions. It simply appears once the model crosses a certain threshold of size and complexity. Skills like multi-step arithmetic, commonsense reasoning, and understanding context in tricky word problems suddenly materialized. Researchers discovered that performance on these tasks didn't improve smoothly as models grew. Instead, it would be flat, near-random, and then suddenly shoot up to impressive accuracy at a specific scale. The ability to perform a chain of thought was one of these emergent skills. The potential was there all along, but it was latent, waiting for the models to become powerful enough to unlock it.
The Discovery That Changed Everything
The 'aha!' moment arrived in 2022 with a landmark paper from Google researchers titled 'Chain-of-Thought Prompting Elicits Reasoning in Large Language Models'. The researchers, including Jason Wei and Denny Zhou, didn't invent a new type of AI. Instead, they discovered how to talk to the new, massive AIs that already existed. They found that by providing just a few examples of step-by-step reasoning in a prompt—or even just by adding the simple phrase 'Let's think step by step'—they could coax these large models into using their emergent reasoning capabilities. The results were stunning. On complex benchmarks like math word problems, accuracy jumped from mediocre to state-of-the-art. It wasn't that the models learned a new skill; the prompt gave them permission and a framework to use a skill they had quietly acquired. This simple prompting technique was the key that unlocked a door no one was sure even existed.
