It still appears in some model reports, but researchers increasingly use harder tests such as MMLU-Pro and Humanity's Last Exam, because top models have neared the ceiling and some MMLU answers are wrong.

MMLU

In short

MMLU (Massive Multitask Language Understanding) is a benchmark of about 16,000 multiple-choice questions across 57 subjects, from maths and history to law. It was the standard test of an AI model's general knowledge from 2020 until top models neared full marks.

Also called: Massive Multitask Language Understanding, MMLU benchmark
Topic: Testing and benchmarks
Level: Some background helps
First described: Introduced by Hendrycks, Burns, Basart, Zou, Mazeika, Song and Steinhardt in September 2020, in "Measuring Massive Multitask Language Understanding"¹.

MMLU was published in September 2020 by Dan Hendrycks and colleagues, then at UC Berkeley¹. It collects 15,908 multiple-choice questions, each with four options, across 57 subjects including elementary maths, US history, computer science and law¹.

The model is scored on the share of answers it gets right. Random guessing gives about 25%. The authors measured untrained crowd workers at 34.5% and estimated expert-level accuracy at about 89.8%¹. When it launched, most models scored near chance, and the largest GPT-3 model reached 43.9%¹.

It became the headline number in model launches. OpenAI reported 86.4% for GPT-4 in March 2023².

MMLU has since lost much of its value at the top end. By early 2025 the authors of Humanity's Last Exam noted that leading models score over 90% on it³. A 2024 audit by a team including University of Edinburgh researchers estimated that 6.49% of its questions contain errors, and found 57% of the virology questions they checked were faulty⁴. Harder successors include MMLU-Pro, which uses ten answer options and drops trivial or noisy questions⁵.

MMLU tests recall and reasoning on exam-style questions. It does not test whether a model can carry out a task, write working code or avoid making things up.

Example

OpenAI also translated MMLU into other languages to test GPT-4, including Welsh, and found GPT-4 beat GPT-3.5's English score in most languages it tried².

Why it matters

For several years MMLU was the single number most often quoted to show an AI model was smarter than the last one. Its saturation is a good example of why one score should not be trusted on its own.

Questions people ask

What does MMLU stand for?

Massive Multitask Language Understanding. It is a multiple-choice test covering 57 subjects¹.

What is a good MMLU score?

Context matters. The original paper put expert-level accuracy at about 89.8% and random guessing at 25%¹. Leading models now score over 90%³, so a high score no longer separates the best models.

Is MMLU still used?

It still appears in some model reports, but researchers increasingly use harder tests such as MMLU-Pro⁵ and Humanity's Last Exam³, because top models have neared the ceiling and some MMLU answers are wrong⁴.

Related terms

Sources

Back to the glossary