The common measure of how fast a model produces text. Useful mainly for comparison: reading speed is roughly ten tokens a second, so anything above that feels acceptable for text a person is reading, while output feeding another program wants to be far quicker. Quoted figures usually assume ideal conditions and a short prompt.
Technical view
Should be reported separately for prefill, processing the prompt, and decode, generating output, since the two differ by orders of magnitude. Per-user throughput falls as batch size rises even while aggregate throughput improves, so a single number without stated batch size, sequence length, quantisation and hardware is close to meaningless. Speculative decoding, quantisation and attention optimisations such as paged and flash attention are the standard levers.
Related
Part of AI Terminology Index, section: Cost, Speed and Service Levels.