Jargonautica
How the Machinery Works

Batch Normalisation

Also called: BatchNorm

Rescaling the numbers passing between layers so they stay in a sensible range. Before it existed, deep networks were fragile and slow to train. It was one of the handful of practical tricks that made depth work at all, though transformers use a different variant.

Technical view

Normalises activations across the batch dimension using batch statistics, with learned scale and shift parameters, and stores running averages for inference. This creates a train-test discrepancy and a dependence on batch size that makes it awkward for small batches, variable-length sequences and distributed training, which is why sequence models moved to layer normalisation instead.


Part of AI Terminology Index, section: How the Machinery Works.

How it connects

Batch Normalisation links to 5 other terms. The faint lines are links between those terms. See it in the full graph.

Neural NetworkNeural NetworkLayer NormalisationLayer NormalisationRegularisation and DropoutRegularisation and …Vanishing and Exploding GradientsVanishing and Explo…Normalisation and StandardisationNormalisation and S…Batch Normalisation