Context window
In short
A context window is the maximum amount of text, measured in tokens, that an AI model can take into account at once, including your prompt, the conversation so far and its own reply. Text beyond that limit has to be dropped or shortened.
- Also called
- Context length, Context size
- Topic
- The basics
- Level
- Some background helps
Anthropic describes the context window as all the text a model can reference when generating a response, including the response itself. It calls it the model's "working memory", separate from the much larger body of data the model was trained on1.
Sizes have grown sharply. GPT-3, in 2020, had a context window of 2,048 tokens2. Meta doubled Llama 2's to 4,096 tokens in 20233. Google says many Gemini models accept 1 million tokens or more4, and Anthropic says Claude's window is up to 1 million tokens, depending on the model1.
In a chat, every message and reply accumulates in the window1. Chat interfaces such as claude.ai can manage it on a rolling "first in, first out" basis, so the oldest material goes first1. Without large windows, Google notes, developers had to drop old messages, summarise content or use retrieval tools to save tokens4.
Bigger is not automatically better. Anthropic warns that as token count grows, accuracy and recall degrade, which it calls context rot1. A 2023 study found models often used information best when it sat at the start or end of a long input, and worst when it was in the middle5.
Example
Pasting a long contract or report into a chatbot only works if it fits in the model's context window. Google says 1 million tokens is roughly 50,000 lines of code or 8 average-length English novels4.
Why it matters
It sets how much you can hand an AI at once, from a long contract to a whole codebase, and helps explain why very long chats can drift.
Questions people ask
What happens when you go over the context window?
Something has to go. Anthropic says chat interfaces such as claude.ai can manage the window on a rolling first-in, first-out basis, so older material drops out1.
Is a bigger context window always better?
No. Anthropic notes accuracy and recall degrade as the token count grows1, and a 2023 study found models struggle to use information placed in the middle of long inputs5.
How big is 1 million tokens?
Google's examples: about 50,000 lines of code, 8 average-length English novels, or transcripts of over 200 average-length podcast episodes4.
Related terms
Tools that use it
- ClaudeAnthropic's chatbot, strong on writing and on pointing out holes in an argument, with a tight free allowance.
- Google GeminiGoogle's chatbot, built into Android, Gmail and Docs, with a generous free plan.
Sources
- Anthropic, Claude Platform Docs, Context windows
- Brown et al. (2020), Language Models are Few-Shot Learners, arXiv:2005.14165
- Touvron et al. (2023), Llama 2: Open Foundation and Fine-Tuned Chat Models, arXiv:2307.09288
- Google AI for Developers, Gemini API docs, Long context
- Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172