Browse tools

How AI models work

Encoder and decoder

In short

An encoder and decoder are the two halves of many AI models: the encoder reads an input, such as a sentence or audio clip, and turns it into numbers; the decoder turns those numbers into an output, such as a translation or transcript.

Also called
Encoder-decoder, Encoder-decoder architecture, Sequence-to-sequence, Seq2seq
Topic
How AI models work
Level
Some background helps
First described
Cho and colleagues named the RNN Encoder-Decoder in June 20141, and Sutskever, Vinyals and Le at Google published sequence-to-sequence learning in September 20142.

The design was built for translation. In 2014 Kyunghyun Cho and colleagues proposed an RNN Encoder-Decoder, in which one recurrent network encodes a sequence of symbols into a fixed-length vector and another decodes that vector into a new sequence1. The same year, Ilya Sutskever, Oriol Vinyals and Quoc Le at Google used one deep LSTM network to map a sentence to a vector and another to decode it, translating English into French2.

The weak point was that single vector. Attention, also from 2014, let the decoder look back over every part of the input instead3. The 2017 transformer kept the encoder-decoder layout but replaced recurrence with attention: the encoder maps the input to a sequence of representations, and the decoder generates the output one symbol at a time4.

Modern models use one half or both. Google's BERT is encoder-only, its name standing for Bidirectional Encoder Representations from Transformers, and it was built for understanding tasks such as question answering5. OpenAI's original GPT was a decoder-only transformer6, the pattern used for text generation. OpenAI's Whisper speech recogniser uses a full encoder-decoder transformer7.

Encoder and decoder here are not encryption terms. They describe which way data flows: into a compact internal form, or back out of it.

Example

When a transcription app runs OpenAI's Whisper on a voice note, the encoder turns the audio into internal representations and the decoder writes out the transcript7.

Why it matters

Knowing which half a model uses tells you what it is for. Encoders are good at understanding input; decoders are good at producing text.

Questions people ask

Is ChatGPT an encoder or a decoder?

The GPT family began as decoder-only. The original 2018 GPT was a 12-layer decoder-only transformer with masked self-attention6. Decoders generate text one token at a time.

What is BERT used for?

BERT is an encoder-only model that reads text in both directions at once5. It is fine-tuned for understanding tasks such as question answering and language inference rather than for writing long replies5.

What is an encoder-decoder model?

A model with two parts: an encoder that turns an input sequence into an internal representation and a decoder that turns that representation into an output sequence1. The original transformer used this layout for translation4.

Related terms

Transformer Attention mechanism Recurrent neural network Speech recognition Latent space Autoregressive model

Tools that use it

Sources

  1. Cho et al. (2014), Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, arXiv:1406.1078
  2. Sutskever, Vinyals and Le (2014), Sequence to Sequence Learning with Neural Networks, arXiv:1409.3215
  3. Bahdanau, Cho and Bengio (2014), Neural Machine Translation by Jointly Learning to Align and Translate, arXiv:1409.0473
  4. Vaswani et al. (2017), Attention Is All You Need, arXiv:1706.03762
  5. Devlin, Chang, Lee and Toutanova (2018), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805
  6. Radford et al. (2018), Improving Language Understanding by Generative Pre-Training, OpenAI
  7. Radford et al. (2022), Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), arXiv:2212.04356

Back to the glossary