Attention mechanism

In short

An attention mechanism is the part of an AI model that decides which other words in a passage matter most for the word it is working on, by giving each one a weight. It is the core of today's chatbots.

Also called: Attention, Self-attention, Multi-head attention
Topic: How AI models work
Level: Some background helps
First described: Bahdanau, Cho and Bengio, September 2014, in "Neural Machine Translation by Jointly Learning to Align and Translate"¹.

Attention was introduced in 2014 for machine translation by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio¹. Earlier translation networks squeezed a whole source sentence into one fixed-length list of numbers, which the authors argued was a bottleneck¹. Their fix let the model search the source sentence for the parts most relevant to each word it was about to write¹.

The 2017 transformer paper gave the version used today. Attention maps a query and a set of key-value pairs to an output, all of them vectors, and the output is a weighted sum of the values, with each weight set by how well the query matches a key². In plain terms, each word asks a question, every other word offers a label, and the best matches get the most say. Self-attention is when a passage attends to itself, and multi-head attention runs several attention steps side by side so the model can track different kinds of relationship at once².

Attention has a cost. The transformer paper lists self-attention as growing with the square of the sequence length², so doubling the length of a prompt roughly quadruples that part of the work. This inefficiency on long sequences is why researchers have built alternatives such as state space models³.

You meet attention in every transformer-based chatbot. OpenAI's first GPT model was a 12-layer decoder-only transformer with masked self-attention heads⁴, and Meta's Llama 3 is a standard dense transformer with a context window of up to 128,000 tokens⁵.

Despite the name, it is not attention in the human sense. It is arithmetic: a fresh set of weights computed for every token.

Example

Type "My sister bought a car last week. Is it electric?" into a chatbot. When the model processes "it", attention gives most weight to "car" and little to "sister" or "week", so the answer is about the car.

Why it matters

Attention is what lets a model use the whole of your prompt rather than just the last few words. Its cost also explains why very long documents are slower and dearer to process.

Questions people ask

Is attention the same as a transformer?

No. Attention came first, in 2014, as an add-on to recurrent translation networks¹. The transformer, from 2017, is a whole architecture built solely on attention, with the recurrence removed².

What are query, key and value in attention?

They are three vectors computed for each token. The query is what a token is looking for, keys are what other tokens offer, and values are the information passed on. Each output is a weighted sum of values, weighted by how well the query matches each key².

Why is attention expensive for long documents?

Every token is compared with every other token, so the work grows with the square of the input length². That is one reason long-context models cost more to run and why alternatives are being explored³.

Related terms

Tools that use it

Sources

Back to the glossary