How much calculation you do for each number you fetch from memory. If it is low, the chip spends its time waiting rather than computing, and the impressive figure on the specification sheet is irrelevant. Almost everything about making inference fast comes down to raising this ratio.
Technical view
Operations per byte of memory traffic, compared against a device’s ratio of peak compute to peak bandwidth in a roofline analysis to determine which limit binds. Matrix-matrix multiplication has high intensity; matrix-vector multiplication, which is what single-sequence decoding performs, has low intensity. Batching converts the latter into the former, which is the fundamental reason serving systems batch aggressively.
Related
Part of AI Terminology Index, section: Making Models Smaller, Faster and Cheaper.