Long short-term memory

In short

Long short-term memory (LSTM) is a type of recurrent neural network built to remember information across long sequences, such as sentences or speech. Small learned switches called gates decide what to store, what to output and what to forget.

Also called: LSTM, LSTM network
Topic: How AI models work
Level: Some background helps
First described: Sepp Hochreiter and Jürgen Schmidhuber, "Long Short-Term Memory", Neural Computation, 1997¹.

Recurrent neural networks read data one step at a time and carry a memory forward. In practice they struggled to learn links between things far apart, because the training signal travelling back through the steps tends to either blow up or fade away¹.

Sepp Hochreiter and Jürgen Schmidhuber proposed LSTM in 1997 to fix this¹. Each unit has an internal memory cell that lets the error signal flow back without shrinking or exploding, plus gate units that learn when to open and close access to it¹. In their tests LSTM could bridge gaps of more than 1,000 time steps¹.

A 2000 paper by Felix Gers, Schmidhuber and Fred Cummins added a "forget gate", so a cell can learn to reset itself instead of letting its memory grow without limit². The standard LSTM layer in the PyTorch library has input, forget and output gates³.

LSTMs powered a jump in machine translation. In September 2016 Google moved Chinese to English in Google Translate onto GNMT, a deep LSTM network with eight encoder and eight decoder layers⁴⁵. Google reported that it cut translation errors by 55% to 85% on several major language pairs, judged by bilingual human raters⁵.

The 2017 transformer design dropped recurrence entirely, which made training more parallel and much faster⁶. LSTM is still a standard building block in machine learning libraries³.

Example

In 2016 Google Translate began handling about 18 million Chinese to English translations a day with GNMT, its LSTM-based system⁵.

Why it matters

LSTM was how neural networks first learned to handle long stretches of language, and it made neural machine translation good enough for an everyday product. It also explains why the transformer's arrival in 2017 mattered so much.

Questions people ask

What is the difference between an LSTM and a transformer?

An LSTM reads a sequence one step at a time and carries a memory forward¹. A transformer drops that step-by-step reading and uses attention to look at the whole sequence at once, which makes training more parallel and faster⁶.

What problem does LSTM solve?

Ordinary recurrent networks find it hard to learn long-range links, because the training signal either explodes or vanishes as it is passed back through time. LSTM's memory cell keeps that signal steady, letting it learn gaps of more than 1,000 steps¹.

Are LSTMs still used?

Yes. LSTM remains a standard layer in libraries such as PyTorch³. Its main rival is the transformer, which trains faster because it processes a whole sequence in parallel rather than step by step⁶.

Related terms

Sources

Back to the glossary