Jargonautica
Hardware, Compute and Training Mechanics

Scaling Laws

Also called: Scaling Law

The observed finding that models get predictably better as you increase their size, the amount of data they see, and the computing power spent on training, all together. This predictability is why organisations have been willing to commit enormous sums to single training runs: the improvement can be forecast reasonably well in advance.

Technical view

Empirical power-law relationships between loss and model size, dataset size and compute. Later work established that earlier models were substantially under-trained on data relative to their parameter count, shifting practice towards smaller models trained on far more tokens. Scaling laws predict loss, which is not the same as predicting downstream capability, and there is active debate about where and whether they flatten out.


Part of AI Terminology Index, section: Hardware, Compute and Training Mechanics.

How it connects

Scaling Laws links to 15 other terms, and the 12 closest are drawn here. The faint lines are links between those terms. See it in the full graph.

ComputeComputeTrainingTrainingLarge Language ModelLarge Language ModelEmergent CapabilitiesEmergent Capabiliti…Artificial General IntelligenceArtificial General …Data Centre and InterconnectData Centre and Int…The Bitter LessonThe Bitter LessonInductive BiasInductive BiasCompute GovernanceCompute GovernanceMixture of ExpertsMixture of ExpertsCompute-optimal TrainingCompute-optimal Tra…Recursive Self-improvementRecursive Self-impr…Scaling Laws