Understanding Its Function First
Adam maintains historical statistics of gradients and adjusts the update scale for different parameters, making it a common gradient optimization algorithm.
Understanding Through an Example
If one parameter's gradient frequently large while another's is frequently small, using the same raw step size for both may not coordinate well. Adam simultaneously considers recent directions and the scale of squared gradients, then computes respective updates for each.
Parameter Change Signal for This Round
Gradient Mean and Gradient Squared Mean
Scale Adjustment per Parameter
A Step Deeper
The first-order moment can be understood as smoothed directional information; the raw second-order moment comes from squared gradients and is used to estimate the scale of change. Initializing statistics to zero creates early bias, which the algorithm corrects. It then uses the corrected first-order moment divided by the square root of the corrected second-order moment (with epsilon) to compute the update.
For a Closer Look
For the mathematical expression, refer to Algorithm 1 in the original paper: m and v respectively accumulate gradients and squared gradients, β₁ and β₂ control how much history to retain, α controls the overall step size, and ε prevents division by too small a number. Understanding the role of each component is more helpful than memorizing the default values.
What It Does Not Tell You
Adam still requires settings like learning rate and does not automatically guarantee optimal results. AdamW decouples weight decay from gradient updates, making it a related variant; they cannot be written as exactly equivalent. Similar naming also does not imply that it belongs to the same level as PPO.
What to Learn Next
References
- Adam: A Method for Stochastic Optimization: Section 2 and Algorithm 1: first and second-order moments, bias correction, and parameter updates.
Reference verified on 2026-09-09; the original paper is used to explain mechanisms, and the examples in the text are for instructional purposes.