AI detector

In short

An AI detector is a tool that estimates whether a person or an AI model wrote a passage of text. It returns a label or a score. That result is a guess, not proof of who wrote it.

Also called: AI detection, AI text detector, GPT detector
Topic: Using AI tools
Level: Beginner
First described: Sebastian Gehrmann, Hendrik Strobelt and Alexander Rush introduced GLTR in June 2019, a tool that highlights statistical traces of model-generated text. In their study it raised people's detection of fake text from 54% to 72%⁴.

A 2023 study of seven widely used detectors says most of them score perplexity: how surprised a language model is by the next word. Wording the model finds easy to predict is treated as more likely to be machine-made, and the authors note that generated text is often easy to predict because the model picks likely words¹.

OpenAI published its own classifier on 31 January 2023. On an English test set it correctly marked 26% of AI-written text as "likely AI-written", and it wrongly marked human text as AI-written 9% of the time. OpenAI said it was impossible to reliably detect all AI-written text, and that the classifier should not be the main way anyone decides who wrote something. It was very unreliable under 1,000 characters, much worse in languages other than English, and unreliable on code. Text can also be edited to evade it. On 20 July 2023 OpenAI took the classifier offline because of its low accuracy².

The same 2023 study tested those seven detectors on 91 essays from the TOEFL English exam, written by people who are not native English speakers, and on 88 essays by US eighth-graders. The tools were accurate on the US essays. On the human TOEFL essays the average false-positive rate was 61.3%: they called human writing AI-written. Asking a chatbot to rewrite AI essays in more literary language drove detection close to zero. The authors warn against using detectors to judge schoolwork¹.

A detector is not a watermark check. The US standards body NIST says synthetic-content detection may look for a mark the maker embedded, such as metadata or a digital watermark, or it may look for other traits in the content itself³. The 2023 study is the evidence for the human cost of the second approach: real essays were called AI-written¹. OpenAI's own note is the evidence that editing can hide AI text from a classifier².

Example

A teacher pastes a student's English essay into a detector before giving a grade. In the 2023 study, human TOEFL essays were labelled AI-written 61.3% of the time on average across seven detectors¹.

Why it matters

A percentage on a detector page can decide a grade. OpenAI has said its own classifier was not reliable enough to be the main way anyone decides who wrote a text².

Questions people ask

Is an AI detector proof that a text was written by AI?

No. OpenAI's classifier caught 26% of AI text in its English test and wrongly flagged human text 9% of the time. OpenAI said it should not be used as the main way to decide who wrote something, and withdrew it in July 2023 because the accuracy was too low².

Why do AI detectors flag non-native English writing?

Many detectors treat predictable wording as a sign of AI. Writers who are not native English speakers often use a narrower range of words, which is also predictable. A 2023 study found an average false-positive rate of 61.3% on human TOEFL essays, against accurate results on essays by US eighth-graders¹.

Is an AI detector the same as a watermark?

No. A detector guesses from the content after the fact. A watermark is a signal put into the content when it is generated, then read by a checker. NIST treats those as related but different techniques³.

Can you trick an AI detector?

Often, yes. OpenAI said AI-written text can be edited to evade a classifier². In the 2023 detector study, asking a chatbot to rewrite its own essays in more literary language dropped detection close to zero¹.

Related terms

Tools that use it

Sources

Back to the glossary