What happens when you actually use a trained model. Cheap and fast compared to training, and this is what runs every time you send a message. Because it happens on every single request, at scale the total cost of inference across a model’s life typically dwarfs the cost of training it.
Technical view
For a language model, inference is autoregressive: one token is generated, appended to the input, and the process repeats, which makes generation inherently sequential and latency-bound. Sampling parameters such as temperature control randomness, with higher values producing more varied and less predictable output. Optimisation techniques include key-value caching to avoid recomputing attention over the whole prompt each step, quantisation to reduce numerical precision, and batching many users’ requests together.
Related
Part of AI Terminology Index, section: Building and Using Models.