The direction of steepest increase, and how steep it is. Training follows it downhill. In one dimension it is just the slope of a line; with a billion parameters it is a list of a billion slopes, one for each, saying how much that particular number is contributing to the error.
Technical view
The vector of partial derivatives of a scalar function with respect to its inputs. For a loss over parameters, it points towards steepest ascent, so updates move against it. Its magnitude drives the vanishing and exploding gradient problems; its noise, from mini-batching, is a form of implicit regularisation that appears to help generalisation rather than merely being a computational compromise.
Related
Part of AI Terminology Index, section: Maths and Statistics.