Benchmark

In short

A benchmark is a fixed, shared test used to measure and compare AI models. Every model answers the same questions or tasks, and the results are scored the same way, so their scores can be compared.

Also called: AI benchmark, LLM benchmark, Benchmark test
Topic: Testing and benchmarks
Level: Beginner
First described: No single inventor. Shared test sets became central to machine learning research through datasets such as ImageNet, published by Deng and colleagues in 2009⁶.

A benchmark is a set of questions or tasks with known correct answers, plus a fixed scoring method. Researchers run a model on it and report a single number, usually the percentage it got right.

MMLU is a typical example. It has 15,908 multiple-choice questions across 57 subjects, from maths to law¹. When OpenAI launched GPT-4, it reported an MMLU score of 86.4%². That is the pattern: a new model arrives with a table of benchmark scores.

Benchmarks are useful because they are repeatable. They are not the same as real-world usefulness. A model can do well on a test without being reliable at the job the test stands in for.

They also wear out. Stanford's AI Index found that tests introduced in 2023 were largely beaten within a year: on the coding benchmark SWE-bench, scores rose from 4.4% in 2023 to 71.7% in 2024³. The 2026 edition says tests meant to last years are now saturated in months⁴.

A second problem is leakage. If test questions end up in a model's training data, its score rises without the model getting more capable. OpenAI stopped reporting one popular coding benchmark in 2026 for this reason⁵.

Example

When a company announces a new chatbot, the launch post usually includes a table comparing it with rivals on tests such as MMLU or SWE-bench. Those numbers are benchmark scores².

Why it matters

Benchmark scores are the main evidence companies use to claim one model is better than another. Knowing what a benchmark does and does not measure helps you read those claims sceptically.

Questions people ask

Are AI benchmarks reliable?

Partly. They are repeatable, but scores can be inflated when test questions leak into training data⁵, and some benchmarks contain wrong answers. Stanford's 2026 AI Index reports invalid question rates from 2% to 42% on widely used tests⁴.

What is the most important AI benchmark?

There is no single one. Companies report many, each covering a different skill: general knowledge (MMLU), coding (SWE-bench), expert science (GPQA) or very hard academic questions (Humanity's Last Exam). Which matters depends on what you want the model to do.

Why do new benchmarks keep appearing?

Because old ones stop separating the best models. Once top scores bunch near the maximum, a test can no longer show progress, so researchers build harder ones³⁴.

Related terms

Sources

Back to the glossary