The Textbook Definition of Weight Decay
First, let's cover the basics you'd find in an AI textbook. Weight decay is a regularization technique used to prevent a model from 'overfitting'. Overfitting is when a model learns the training data so perfectly that it also learns all the noise and
irrelevant details, causing it to perform poorly on new, unseen data. To fight this, weight decay penalizes large weights in the neural network. The idea is that a model with smaller, more distributed weights is simpler and more likely to generalize well to new situations. In academia, this is almost always described as being equivalent to L2 regularization. L2 regularization works by adding a penalty term to the model's loss function—the metric it's trying to minimize. This penalty is proportional to the sum of the squared values of all the weights in the model, effectively telling the optimizer to keep the weights small while it learns.
Theory vs. Practice: The Big Disconnect
For a long time, the terms 'weight decay' and 'L2 regularization' were used interchangeably, and for good reason: with a basic optimizer like stochastic gradient descent (SGD), they are mathematically identical. Applying L2 regularization results in an update rule that subtracts a small fraction of each weight during every training step—in other words, the weights decay. This equivalence led library developers to implement weight decay simply by adding an L2 penalty. However, modern deep learning rarely uses basic SGD. Instead, practitioners favor more advanced 'adaptive' optimizers like Adam, which adjust the learning rate for each parameter individually based on the history of its gradients. And this is where the textbook theory breaks down. When you combine L2 regularization with an optimizer like Adam, the two concepts are no longer the same. The regularization term gets mixed in with the gradient calculations and is then affected by Adam's adaptive scaling, which means the 'decay' is no longer uniform across all weights.
Why the Difference Matters So Much
This subtle implementation detail has significant consequences. With L2 regularization, weights that have large historical gradients (a sign that they are changing a lot) receive a smaller regularization effect than weights with small gradients. This is the opposite of what you might want. The true 'weight decay' method, on the other hand, should apply a small, consistent shrinkage to every single weight, regardless of its gradient history. This disconnect was famously highlighted in a 2017 paper by Ilya Loshchilov and Frank Hutter titled 'Decoupled Weight Decay Regularization'. They demonstrated that using L2 regularization with Adam often led to worse performance than using good old SGD with momentum, especially in tasks like image classification. The 'wrong' type of decay was holding back the more advanced optimizer.
AdamW: The Decoupled Solution
The solution proposed by Loshchilov and Hutter was simple but powerful: decouple the weight decay from the gradient update. Instead of adding a penalty to the loss function (L2 regularization), they implemented weight decay as it was originally intended—a direct subtraction from the weights after the main optimization step. This new version of the Adam optimizer was dubbed 'AdamW'. In AdamW, the main gradient-based update happens, and then, as a separate step, every weight is slightly reduced. This ensures that the decay effect is consistent and not influenced by the adaptive learning rates. The fix worked wonders, substantially improving Adam's ability to generalize and allowing it to compete with, and often outperform, SGD on a wider range of tasks. Today, AdamW is a standard optimizer available in all major deep learning frameworks like PyTorch and TensorFlow.
