Training a small model to imitate a large one. The big model answers a great many questions, and the small model learns to produce the same answers. The result is a cheaper, faster model that retains much of the larger one’s ability on the tasks it was distilled for. Most of the small, fast models offered commercially are produced this way.
Technical view
A student model is trained on the teacher’s full output distribution rather than only its final answers, since the relative probabilities across all options carry more information than the single top choice. Distillation transfers capability efficiently but tends to narrow it: students typically match teachers on the distribution they were distilled over and fall away faster on unusual or out-of-distribution inputs. Many model providers explicitly forbid using their outputs to train competing models.
Related
Part of AI Terminology Index, section: Making Models Smaller, Faster and Cheaper.