A token is the smallest processing unit into which a language model breaks down text. Before processing, a tokenizer splits input text into these units — depending on the method, whole words, word fragments (subwords) or individual characters. Modern models such as the GPT family typically use subword methods like byte-pair encoding, which merges frequent character sequences into a single token.
Each unique token is assigned an ID; the set of all known tokens forms the model's vocabulary. Text thus becomes a sequence of token IDs on which the model computes and for which it predicts the next token. As a rule of thumb, one token corresponds roughly to four characters, or about 0.75 words, in English.
Tokens are also the unit for limits and billing: the size of the context window is measured in tokens, and the cost of commercial models is usually based on the number of input and output tokens.