Browse tools

How AI models work

Transformer

In short

A transformer is the neural network design behind almost every modern AI chatbot. It reads a whole passage at once and uses a technique called attention to work out which words matter most to each other.

Also called
Transformer architecture, Transformer model
Topic
How AI models work
Level
Some background helps
First described
Vaswani and colleagues at Google, June 2017, in "Attention Is All You Need".

Before 2017, language models mostly read text one word at a time, in order. That made them slow to train and bad at linking words that sat far apart in a sentence.

The transformer, introduced by eight Google researchers in the paper "Attention Is All You Need", dropped that word-by-word reading entirely1. Instead, every word looks at every other word at the same time and decides how much weight to give each one. That step is called attention.

Because the work happens in parallel, transformers train far faster on modern chips. The original paper reported state-of-the-art translation results after 3.5 days of training on eight GPUs1.

The design scaled. The "GPT" in ChatGPT stands for generative pre-trained transformer2, and Meta's Llama models are transformers too3.

Example

In the sentence "The trophy didn't fit in the suitcase because it was too big", attention is how the model links "it" to "trophy" rather than "suitcase".

Why it matters

It is the single idea most of today's AI boom rests on. When people talk about large language models, they almost always mean transformers.

Questions people ask

Who invented the transformer?

Eight researchers, most of them at Google: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin. Their paper was posted in June 20171.

What does the T in ChatGPT mean?

Transformer. GPT stands for generative pre-trained transformer, the name OpenAI gave its model family in 20182.

Are all AI models transformers?

No. Many image generators use diffusion models, and some newer language models use state space designs. But GPT and Llama, two of the best-known model families, are transformers23.

Related terms

Attention mechanism Large language model Next-token prediction State space model Positional encoding Mixture of experts

Tools that use it

Sources

  1. Vaswani et al. (2017), Attention Is All You Need, arXiv:1706.03762
  2. Radford et al. (2018), Improving Language Understanding by Generative Pre-Training, OpenAI
  3. Touvron et al. (2023), LLaMA: Open and Efficient Foundation Language Models, arXiv:2302.13971

Back to the glossary