Doing most of the arithmetic in a lower-precision number format to go faster and use less memory, while keeping the sensitive parts in full precision. It roughly doubles training speed on modern hardware for almost no loss in quality, which is why everyone does it.
Technical view
Stores and computes in 16-bit formats while maintaining a full-precision copy of weights and performing accumulations in higher precision. FP16 requires loss scaling to avoid gradient underflow; BF16 has the same exponent range as FP32 and avoids the issue, which is why it is now preferred. FP8 training is emerging on newer hardware with additional scaling machinery.
Related
Part of AI Terminology Index, section: How the Machinery Works.