GPQA
In short
GPQA is a benchmark of graduate-level multiple-choice questions in biology, physics and chemistry, written by domain experts. Non-experts who could search the web, but not use another model, still scored 34%, which is why the authors call the questions Google-proof.
- Also called
- Graduate-Level Google-Proof Q&A, GPQA benchmark
- Topic
- Testing and benchmarks
- Level
- Technical
- First described
- Rein and colleagues introduced GPQA in a paper submitted on 20 November 2023, GPQA: A Graduate-Level Google-Proof Q&A Benchmark1.
GPQA stands for Graduate-Level Google-Proof Q&A. It is a set of graduate-level multiple-choice questions in biology, physics and chemistry, written by domain experts1. The main set has 448 questions, drawn from an extended set of 5461.
On the extended set, experts reached 65% accuracy and non-experts reached 34%1. Those non-experts had, or were studying for, a PhD in another field. They could use the open web, but not a language-model assistant, and they spent 37 minutes on average on each question1. Each question has four options, so guessing scores 25%1. The authors call the questions Google-proof because of that result1. Counting clear mistakes that experts later identified, the authors estimate that 74% of extended-set questions have an uncontroversial correct answer1.
On the main set of 448, the same table lists expert accuracy at 71.9% and non-expert accuracy at 30.4%. The authors say those two figures are skewed by how the main set was selected1. Models are scored by accuracy, the share of questions answered with the correct option. In the paper, a GPT-4 baseline with few-shot chain-of-thought prompting reached 39%1.
The authors present GPQA as a setting for scalable oversight experiments, in which people supervise systems on questions that are hard even for skilled non-experts1. They also state limits: the set is small and not ideal for training, and they make no claim that it represents the questions that come up in scientific work1. A later exam, Humanity's Last Exam, uses 2,500 questions across over a hundred subjects, including mathematics, humanities and the natural sciences2.
Example
A lab scores a model on GPQA by the share of questions where it picks the correct option. In the original paper, random guessing on the four-option questions scores 25%, and a GPT-4 chain-of-thought baseline reached 39%1.
Why it matters
GPQA is a hard test of specialist science questions that stayed difficult for PhD-level non-experts who could search the web1. The authors offer it for oversight research, and they do not claim the questions represent scientific practice1.
Questions people ask
What does GPQA stand for?
GPQA stands for Graduate-Level Google-Proof Q&A, the name Rein and colleagues gave the benchmark in their 2023 paper1.
What subjects does GPQA cover?
The questions are graduate-level multiple-choice items in biology, physics and chemistry, written by domain experts. The main set has 448 questions1.
Why is GPQA called Google-proof?
Non-experts with PhD training in other fields could search the web, but not use a language-model assistant, and still reached 34% after 37 minutes per question on average. Experts in the field reached 65% on that extended set1.
Does a GPQA score measure scientific ability in general?
The paper presents the score as accuracy on this question set, for scalable oversight experiments. The authors say they do not claim the questions represent those that arise in scientific practice1.