Browse tools

How AI models work

Autoregressive model

In short

An autoregressive model generates a sequence one piece at a time, predicting each new piece from all the pieces it has already produced. Chatbots such as ChatGPT work this way, writing one token after another.

Also called
Autoregressive, AR model, Autoregression
Topic
How AI models work
Level
Some background helps
First described
George Udny Yule, 1927, modelling sunspot numbers in the Philosophical Transactions of the Royal Society2. The approach is now called the autoregressive model3.

Autoregressive means the model regresses on itself: its own earlier outputs become inputs for its next prediction. The 2017 transformer paper describes its decoder as auto-regressive, "consuming the previously generated symbols as additional input when generating the next"1.

The idea comes from statistics. In 1927 the statistician George Udny Yule modelled Wolfer's sunspot numbers2, predicting each year's value from the values of earlier years, the approach now known as an autoregressive or AR model3.

In AI the same principle drives text, audio and images. OpenAI described GPT-3 as "an autoregressive language model with 175 billion parameters"4. DeepMind's WaveNet generated raw audio autoregressively, with each audio sample conditioned on all previous ones5, and PixelRNN generated images by predicting pixels in sequence6.

The trade-off is speed. Because each step depends on the one before, output has to be produced in order. Google DeepMind notes that autoregressive language models generate text one word or token at a time, and contrasts them with diffusion models, which refine noise step by step7.

Autoregressive describes how output is produced, not the internal wiring. A transformer decoder1 and the recurrent PixelRNN6 are both autoregressive.

Example

Ask a chatbot to finish "Fish and". It predicts "chips", appends it, then predicts the next token using "Fish and chips" as its new input, and so on until it decides to stop.

Why it matters

It explains why AI replies appear word by word and why longer answers take longer: each token has to wait for the one before it.

Questions people ask

Is ChatGPT an autoregressive model?

The GPT models behind it are built that way. OpenAI described GPT-3 as an autoregressive language model4, meaning it writes one token at a time, feeding each new token back in to predict the next.

What is the alternative to autoregressive generation?

Diffusion is the main one. Instead of writing token by token, a diffusion model refines noise step by step. Google DeepMind's experimental Gemini Diffusion applies this to text7.

What does autoregressive mean in statistics?

A model that predicts the next value in a time series from its own previous values. Yule introduced the idea in 1927 using yearly sunspot counts23.

Related terms

Next-token prediction Transformer Diffusion model Large language model Encoder and decoder Inference

Tools that use it

Sources

  1. Vaswani et al. (2017), Attention Is All You Need, arXiv:1706.03762
  2. Yule (1927), On a Method of Investigating Periodicities in Disturbed Series, with Special Reference to Wolfer's Sunspot Numbers, Philosophical Transactions of the Royal Society A 226 (publisher page blocked automated access; metadata via Crossref)
  3. UC Berkeley Stat 153 (Fall 2025), Lecture 16 notes: AR models and the sunspots data
  4. Brown et al. (2020), Language Models are Few-Shot Learners, arXiv:2005.14165
  5. van den Oord et al. (2016), WaveNet: A Generative Model for Raw Audio, arXiv:1609.03499
  6. van den Oord, Kalchbrenner and Kavukcuoglu (2016), Pixel Recurrent Neural Networks, arXiv:1601.06759
  7. Google DeepMind, Gemini Diffusion model page

Back to the glossary