The Paper: One Sample at a Time
In its purest, textbook form, Stochastic Gradient Descent is beautifully simple. To train a model, you pick a single, random data point, calculate how wrong the model's prediction is (the gradient of the loss), and nudge the model's parameters in the right
direction. Repeat this for millions of individual data points, and the model slowly learns. The 'stochastic' part refers to this randomness of picking one sample, which introduces noise that can surprisingly help the model avoid getting stuck in suboptimal solutions. The process is iterative and, in theory, computationally light for any single step.
The Practice: Mini-Batches Rule Everything
In the real world, almost no one uses a single data point per update. Instead, practitioners use 'mini-batches'—small groups of samples (typically 32 to 256) chosen at random. The gradients are averaged across the mini-batch before updating the model. This has two huge practical advantages. First, it smooths out the noisy updates, leading to more stable and reliable convergence. Second, modern hardware like GPUs is highly optimized for parallel matrix operations, making it far more efficient to process a small batch at once than to process single samples sequentially.
The Paper: A Fixed, Tiny Learning Rate
Academic papers often describe the 'learning rate'—the size of the step you take with each update—as a small, constant value that you just have to choose correctly. This simplifies the math and allows for clean proofs of convergence. The idea is that if the steps are small enough, you'll eventually find your way to the lowest point of the error landscape, like a hiker carefully descending a mountain in the fog.
The Practice: The Art of the Learning Rate Schedule
In practice, a fixed learning rate is rarely optimal. Getting it right is a major challenge; too high and the training can diverge, too low and it can take forever. As a result, engineers use 'learning rate schedulers'. These are strategies that adjust the learning rate during training. A common approach is to start with a larger learning rate for faster initial progress and then gradually decrease it to allow for fine-tuning as the model gets closer to a good solution. Some advanced techniques, like 'cosine annealing' or '1cycle' policies, even have the learning rate go up and down to help escape tricky parts of the error landscape.
The Paper: The Simple Vanilla Update
The foundational SGD update rule is straightforward: move the parameters in the opposite direction of the current gradient. It's an elegant concept, but it has a known weakness for navigating long, narrow valleys in the error landscape, where it tends to oscillate back and forth instead of moving directly toward the minimum. While sufficient for many convex problems, it can be inefficient for the complex, non-convex landscapes of deep neural networks.
The Practice: Give it Momentum, Give it Wings
To solve this, practitioners almost always add 'momentum' to SGD. Momentum introduces a velocity component, where the update is a combination of the current gradient and the previous update's direction. This helps the optimizer build up speed in consistent directions and dampens oscillations, allowing it to blast through ravines and settle into minima more quickly. But it doesn't stop there. The most popular optimizers today, like Adam (Adaptive Moment Estimation), are even more sophisticated. Adam not only incorporates momentum but also adapts the learning rate for each individual parameter, making it more robust and often faster to converge than SGD with momentum alone.
