Recurrent neural network
In short
A recurrent neural network is a neural network with a loop. It takes a sequence one step at a time and feeds its own earlier state back in, so later steps can depend on what it has already seen.
- Also called
- RNN, Simple recurrent network
- Topic
- How AI models work
- Level
- Some background helps
- First described
- Elman (1990) traces recurrent links, which give a network a dynamic memory, to Michael I. Jordan's 1986 proposal. Elman's paper is the standard account of the simple recurrent network.1
Many neural networks map an input to an output in one pass. A recurrent network adds a loop. In 1990 Jeffrey Elman described a simple form of that loop: the pattern on the hidden units is copied and fed back in with the next input. He wrote that this gives the network a dynamic memory, and he traced those recurrent links to a 1986 proposal by Michael Jordan.1
The loop is what makes a sequence tractable. Elman presented letters one at a time and trained the network to predict the next letter. The saved hidden state, held in what he called context units, arrived alongside each new letter. His other tests ran from a temporal version of XOR to finding syntactic and semantic features of words.1
The weakness is a long gap. In 1997 Sepp Hochreiter and Jürgen Schmidhuber wrote that learning to store information over extended intervals with recurrent backpropagation takes a very long time, mostly because the error signal decays. Their long short-term memory (LSTM) method could bridge gaps of more than 1,000 time steps, and in their comparisons it beat Elman nets among other recurrent methods.2
On sequence tasks, recurrence was the dominant design until attention. The 2017 transformer paper says the dominant sequence models were complex recurrent or convolutional networks, and that a recurrent layer needs a run of sequential steps that grows with the length of the input. The transformer drops the loop and processes the sequence in parallel.3
A recurrent network is not a transformer, and it is not a convolutional network. Those designs do not pass a hidden state forward one step at a time.3 An LSTM is a recurrent network with a fix for the fading signal, not a separate kind of model.2
Example
Elman showed his network one letter at a time and asked it to predict the next letter. Context units carried the previous hidden state in with each new letter, so the guess could depend on the letters already seen.1
Why it matters
It was the standard way to model a sequence, such as text, before the transformer. The transformer removes the loop so the whole input can be processed in parallel.3
Questions people ask
Is an LSTM the same as a recurrent neural network?
An LSTM is a type of recurrent network. Hochreiter and Schmidhuber introduced it in 1997 so a network could hold information across much longer gaps. They found that a plain recurrent net loses its learning signal as the gap grows.2
Is a transformer a recurrent neural network?
No. Vaswani and colleagues designed the transformer to do without recurrence. A recurrent layer walks a sequence in order. The transformer uses attention and handles the sequence in parallel.3
Why do recurrent networks struggle with long sentences?
The error signal decays as it is sent back through many steps, so the network is slow to learn a link across a long gap. Hochreiter and Schmidhuber called this decaying error flow the reason recurrent backpropagation takes so long.2