A measure of uncertainty. A fair coin has more entropy than a biased one because you can say less about what comes next. Information is what reduces it. This is the basis for compression, for how models are trained, and for the argument that prediction and understanding are more closely related than they first appear.
Technical view
Shannon entropy is the expected number of bits needed to encode outcomes from a distribution. Cross-entropy loss measures the cost of encoding the true distribution using the model’s predicted one, which is why it is the standard training objective for classification and language modelling. Perplexity is its exponential. Decision trees split on information gain, the reduction in entropy achieved by a split.
Related
Part of AI Terminology Index, section: Maths and Statistics.