Perplexity (metric)
In short
Perplexity (metric) is a score for how well a language model predicts a test text. Paloma defines it as the exponential of the average negative log probability of the observed tokens, so a lower score means a higher probability on that text.
- Also called
- PPL, language-model perplexity
- Topic
- Testing and benchmarks
- Level
- Some background helps
- First described
- Jelinek, Mercer, Bahl and Baker introduced perplexity in 1977 as a measure of how difficult a speech-recognition task is. They defined it so that its logarithm is the entropy1.
In 1977, Jelinek, Mercer, Bahl and Baker argued that vocabulary size and branching factors were poor measures of how hard a speech-recognition grammar is. They proposed perplexity instead. On their definition, the logarithm of perplexity is the entropy, and the score is a measure of equivalent choice. They also said the measure has weaknesses1.
For language models, Paloma uses perplexity per token. Add up the natural log of the probability the model gives each token, given the tokens before it, and divide by the number of tokens. Perplexity is e raised to the power of minus that average2. A higher probability for the whole test sequence makes the score lower2.
Paloma says a fair comparison is hard. Vocabulary sets which sequences are possible and whether two perplexity numbers can be compared. The authors also ask for the same evaluation code, a fixed training order, and removal from training of any text that overlaps the evaluation text2. When the vocabulary cannot be shared, they report bits per byte instead of perplexity2.
The benchmark scores fit on 546 English and code domains, rather than one held-out corpus, because a single perplexity does not show whether fit on one kind of text carries over to others2.
Example
Two language models are scored on the same held-out paragraph with the same tokenisation. The model that assigns higher probability to the paragraph has the lower perplexity2.
Why it matters
Perplexity turns a model's token probabilities on a test text into one number2. It compares predictive fit when the text and the calculation stay fixed, and it mixes in the setup when they do not2.
Questions people ask
Is lower perplexity better?
On the same test text, with the same tokenisation and formula, a lower score means the model assigned higher probability to the observed sequence2.
Can perplexity scores be compared across models?
Only under shared rules. Paloma says vocabulary decides whether perplexity numbers are comparable, and it asks for one evaluation format. If the vocabulary must differ, the authors use bits per byte instead2.
What did the 1977 paper use perplexity for?
Jelinek and colleagues used it to measure the difficulty of speech-recognition tasks. They argued it was a better measure of equivalent choice than vocabulary size or a branching factor1.