The component that decides how to apply each update during training. Plain gradient descent takes a fixed-size step downhill; better optimisers adapt the step for each parameter based on its recent history, which makes training faster and much less fussy to configure.
Technical view
Adam and its weight-decay-corrected variant AdamW are the practical default, maintaining per-parameter estimates of the first and second gradient moments. Momentum accelerates along consistent directions and damps oscillation. The optimiser state is typically twice the parameter count in memory, which is a major constraint on large-scale training and the target of sharding techniques. Newer optimisers claim faster convergence but adoption is cautious given the cost of a failed run.
Related
Part of AI Terminology Index, section: How the Machinery Works.