The Old Way: Counting by Epochs
To understand where we are, we have to understand where we started. In traditional machine learning, researchers often talked about 'epochs.' Think of an epoch as a single complete pass through a textbook. If you read every page in a biology book once,
you have completed one epoch. If you go back and read it all again, that is two epochs. In this old-school model, the goal was often to repeat the data until the model 'got it.' It was a predictable, rhythmic way to measure progress: you knew exactly how much information the model had consumed and how many times it had seen each specific fact.
The Great Token Shift
As we moved from simple models to the massive Large Language Models (LLMs) like GPT-3 or Llama 3, the concept of an epoch started to break down. We stopped talking about 'books' and started talking about 'tokens'—the tiny fragments of text that make up words. Modern training isn't about reading a set of books multiple times; it is about swallowing an ocean of data. When you are dealing with trillions of tokens, the idea of a 'complete pass' becomes almost meaningless. For instance, when Meta trained Llama 3, they used over 15 trillion tokens. At that scale, researchers aren't checking off chapters in a book; they are managing a continuous stream of information.
Why More Isn't Always Better
You might think that more data always equals more intelligence, but there is a catch: repetition loses value fast. If you read the same page in that biology textbook fifty times, you aren't learning more biology; you are just memorizing the ink on the page. This is exactly what happens in AI training. If a model sees the same data too many times, it stops learning patterns and starts memorizing specific sequences. This is why modern AI labs spend massive amounts of money and effort on deduplication—cleaning out repetitive text—to ensure every token provides something fresh. The goal is often to keep the model seeing new things, effectively keeping it at less than one full 'epoch' of any single piece of data.
The Power of the Batch
This brings us to 'batch learning,' the quiet engine behind modern AI capability. Instead of updating a model after every single piece of information (which is too chaotic) or waiting until it has seen everything (which is too slow), researchers use batches. A batch is a manageable chunk of data processed all at once to calculate a single update to the model's internal weights. The real magic happens in how large these batches are. Early on in training, when the data is diverse and confusing, large batches help smooth out the noise. As training progresses and the model gets smarter, researchers can scale these batches even further to squeeze out those final, fine-grained details that turn a good model into a great one.













