Changing the step size as training progresses. Start too fast and the model lurches around; stay fast at the end and it never settles. The standard shape is a short ramp up followed by a long decay, and getting it wrong is one of the most common causes of a training run failing.
Technical view
Typical schedules combine linear warm-up over the first fraction of steps with cosine or linear decay to a small final value. Warm-up matters because early gradients are large and adaptive optimiser statistics are unreliable. The peak learning rate is the single most sensitive hyperparameter, scaling with batch size, and schedules must be planned against the total step count in advance, which complicates deciding to train for longer partway through.
Related
Part of AI Terminology Index, section: How the Machinery Works.