First, What Was the Big Idea?
When the Adam (Adaptive Moment Estimation) optimizer was introduced in 2015, it was a game-changer. It cleverly combined two powerful ideas: momentum (using a moving average of past gradients to keep updates moving in a consistent direction) and adaptive
learning rates (giving each parameter its own learning rate that adjusts based on past squared gradients). This made it faster and more reliable than many existing optimizers, quickly becoming the default choice for training deep neural networks. The original paper laid out the math for how these two 'moments' would be estimated and used to update a model's weights. On paper, it was an elegant solution to complex optimization problems. But theory rarely survives contact with reality unchanged.
The Problem with 'Regular' Regularization
The most significant difference between the Adam in the paper and the Adam in your code revolves around a technique called L2 regularization. This method penalizes large weights in a model to prevent overfitting. The standard way to do this is to add a penalty term to the loss function. While this works perfectly fine with simpler optimizers like Stochastic Gradient Descent (SGD), it interacts poorly with Adam. Adam's adaptive learning rates scale updates for each parameter differently. When the L2 penalty is mixed into the gradient calculation, this adaptive scaling distorts the regularization effect. Weights that have large historical gradients get regularized less than intended, while others get regularized more. The result is unpredictable and often less effective regularization.
Enter AdamW: Decoupling Weight Decay
The solution, proposed in a 2019 paper by Ilya Loshchilov and Frank Hutter, was brilliantly simple: decouple weight decay from the gradient update. This led to the creation of AdamW, where 'W' stands for 'Weight Decay'. Instead of mixing the L2 regularization penalty into the loss function, AdamW applies the weight decay directly to the weights after the main Adam update step. This small change has a huge impact. It ensures that the regularization strength is uniform and predictable, just as the practitioner intended. It separates the concerns of choosing a learning rate from choosing a regularization strength, making tuning both much easier and more stable. This decoupled approach proved so much more effective, especially for large models like transformers, that AdamW is now the standard implementation in frameworks like PyTorch and Keras.
Other Subtle, But Important, Tweaks
While decoupled weight decay is the headline change, it's not the only difference. The original paper also introduced bias-correction terms to account for the fact that the moving averages start at zero. In practice, the implementation of these corrections and the handling of the tiny 'epsilon' value (used to prevent division by zero) are critical for numerical stability, especially in the early stages of training. Different deep learning libraries have historically even made slightly different choices in the exact order of operations, such as when to take the square root versus when to add epsilon, further highlighting the gap between a paper's formula and a battle-tested software implementation.
