Jargonautica
Hardware, Compute and Training Mechanics

Epoch, Batch and Learning Rate

Also called: Epoch, Batch Size, Learning Rate

Three basic settings that govern how training runs. An epoch is one complete pass through the training data. A batch is the group of examples processed together before the model updates itself. The learning rate is how big each adjustment is. Getting these wrong is one of the most common reasons training fails.

Technical view

These are core hyperparameters. Large models are often trained for a single epoch or less over their corpus, since data volume exceeds what repeated passes would usefully add and repetition risks memorisation. Batch size trades gradient noise against hardware efficiency. Learning rate schedules, typically a warm-up followed by decay, are standard, and learning rate is generally regarded as the single most consequential hyperparameter to get right.


Part of AI Terminology Index, section: Hardware, Compute and Training Mechanics.

How it connects

Epoch, Batch and Learning Rate links to 7 other terms. The faint lines are links between those terms. See it in the full graph.

TrainingTrainingGradient DescentGradient DescentOverfittingOverfittingWeights, Parameters and BiasesWeights, Parameters…HyperparametersHyperparametersGradient AccumulationGradient Accumulati…Learning Rate ScheduleLearning Rate Sched…Epoch, Batch and Learning Rate