Next-token prediction
In short
Next-token prediction is the task at the heart of AI chatbots: given some text, the model works out which token (a word or part of a word) is most likely to come next, adds one, and repeats until the reply is finished.
- Also called
- Next-word prediction, Language modelling objective, Causal language modelling
- Topic
- How AI models work
- Level
- Beginner
A large language model is trained on huge amounts of text with one simple goal: predict the next token from the ones before it. OpenAI's first GPT paper writes this as maximising the probability of each token given the tokens that precede it1. At the end of each step, the transformer turns its internal numbers into "predicted next-token probabilities" across its whole vocabulary2.
The idea is old. In 1951 Claude Shannon estimated how predictable English is through experiments in which people guessed the next letter of a text when the preceding text was known3. In 2003 Yoshua Bengio and colleagues trained a neural network to learn the probability of word sequences4.
Scale turned this narrow task into something general. GPT-3, a 175 billion parameter model trained to predict text, could translate, answer questions and do 3-digit arithmetic from a few examples in the prompt, with no retraining5. Google reported that Gemini 1.5 kept improving at next-token prediction across contexts of at least 10 million tokens6.
The model does not always pick the single most likely token. Settings such as temperature and top-p control how much randomness goes into each choice7.
Next-token prediction is not fact lookup. The model produces likely text, which is usually but not always true. That gap is the root of hallucinations.
Example
Type "The capital of France is" and the model scores every token it knows. "Paris" scores far higher than "Leeds", so that is almost always what comes next. The model then adds "Paris" to the text and predicts again.
Why it matters
It explains both what chatbots are good at and why they can be confidently wrong. They are built to continue text plausibly, not to check it.
Questions people ask
Do chatbots just predict the next word?
At the point of writing, yes: they produce one token at a time, each predicted from what came before1. But training on that task at scale gave models like GPT-3 broad abilities, such as translation and simple arithmetic, that were never programmed directly5.
Is next-token prediction the same as phone autocomplete?
It is the same basic idea, a guess at what comes next, at a very different scale. GPT-3 had 175 billion parameters and could take in instructions and examples in its prompt5.
Why does a chatbot give different answers to the same question?
Because it usually samples from its predicted probabilities rather than always taking the top token. Temperature and top-p settings control how adventurous that sampling is7.
Related terms
Tools that use it
- ChatGPTOpenAI's general chatbot, the most used AI app in the UK and worldwide.
- Google GeminiGoogle's chatbot, built into Android, Gmail and Docs, with a generous free plan.
- Meta Llama modelsMeta's family of downloadable large language models, from Llama 2 to the multimodal Llama 4
Sources
- Radford et al. (2018), Improving Language Understanding by Generative Pre-Training, OpenAI
- Vaswani et al. (2017), Attention Is All You Need, arXiv:1706.03762
- Shannon (1951), Prediction and Entropy of Printed English, Bell System Technical Journal 30(1)
- Bengio, Ducharme, Vincent and Jauvin (2003), A Neural Probabilistic Language Model, Journal of Machine Learning Research 3
- Brown et al. (2020), Language Models are Few-Shot Learners, arXiv:2005.14165
- Gemini Team, Google (2024), Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv:2403.05530
- OpenAI API reference, Create a model response (temperature and top_p parameters)