A matrix is a grid of numbers; a tensor is the same idea extended to more dimensions. Nearly all the computation in a neural network is multiplying these together, billions of times. This is why graphics chips are so well suited to the work: they were already built to do exactly this for rendering.
Technical view
A matrix is a two-dimensional array representing a linear map; a tensor generalises to arbitrary rank, and in practice the word usually just means a multi-dimensional array with a shape and a data type. Shapes are the source of most implementation errors. Batch dimensions, broadcasting rules and memory layout determine performance as much as the arithmetic does, since matrix multiplication is memory-bound at scale.
Related
Part of AI Terminology Index, section: Maths and Statistics.