In AI, inference refers to the phase in which an already trained model is applied to new, unseen data to produce an output — such as a prediction, classification or generated text. It contrasts with the training phase, in which the model learns from data and adjusts its parameters. Inference is thus the production use of a model: the trained model is deployed and applied to real requests.
For a language model, every response to a prompt is an inference operation: the model processes the input tokens and predicts the output tokens step by step. Unlike the one-time, compute-intensive training, inference happens millions of times in ongoing operation.
For this reason, latency, throughput and cost per request are decisive at inference time. A large share of the operating cost of production AI systems is due to inference, which is why it is accelerated with specialized hardware (GPUs, AI accelerators) and optimizations.