Token
In short
A token is the small chunk of text an AI language model reads and writes, often a short word or part of a longer one. Models count, limit and price their work in tokens rather than words.
- Also called
- Tokens, AI token
- Topic
- The basics
- Level
- Beginner
- First described
- Rico Sennrich, Barry Haddow and Alexandra Birch, August 2015, "Neural Machine Translation of Rare Words with Subword Units", which introduced subword tokens built with byte pair encoding3.
Language models do not read letters or words directly. Text is first split into tokens, which can be single characters, whole words like "cat", or pieces of longer words1. The full set a model uses is its vocabulary, and the splitting step is called tokenisation1.
Google says that for Gemini models a token is about 4 characters, and 100 tokens is about 60 to 80 English words1. Anthropic says the ratio for Claude depends on the model and the language, and that from Claude 4.7 a newer tokeniser produces about 30% more tokens for the same text2.
Splitting words into sub-word pieces lets a model handle rare and new words. A 2015 paper by Rico Sennrich, Barry Haddow and Alexandra Birch encoded rare words as sequences of subword units using byte pair encoding3. GPT-3 later used byte-pair-encoded tokens too4.
Tokens matter in practice. The cost of a Gemini API call depends partly on its input and output tokens1, and a model's context window, the amount it can consider at once, is measured in tokens2. Tokens also explain some odd failures: the GPT-3 authors noted that tasks needing character-level changes were hard because the model sees tokens, not individual letters4.
This kind of token has nothing to do with crypto tokens or login tokens.
Example
Ask a chatbot to spell a long word backwards and it may slip, partly because it sees the word as a few tokens rather than as separate letters4.
Why it matters
Tokens decide what using AI costs and how much it can read at once, so prices and limits are quoted in them.
Questions people ask
How many words is 1,000 tokens?
Roughly 600 to 800 English words for Gemini models, where 100 tokens is about 60 to 80 words1. Other models split text differently, so the ratio varies by model and language2.
Why do AI companies charge per token?
Tokens measure how much text a model reads and writes. For the Gemini API, the cost of a call depends in part on the number of input and output tokens1.
Is a token the same as a word?
No. A token can be a single character, a whole word or part of a longer word1.