Tokeniser
In short
A tokeniser is the tool that chops text into tokens, the small chunks (whole words, parts of words or single characters) that an AI language model actually reads and writes. Each token is then turned into a number.
- Also called
- Tokenizer, Tokenisation, Tokenization, Byte pair encoding, BPE
- Topic
- How AI models work
- Level
- Beginner
- First described
- Sennrich, Haddow and Birch at the University of Edinburgh adapted byte pair encoding, a compression method described by Philip Gage in 1994, for neural machine translation in 20152.
Language models do not see letters or words. As the guide to OpenAI's tiktoken tokeniser puts it, they "see a sequence of numbers (known as tokens)"1. The tokeniser turns text into those numbers on the way in and back into text on the way out.
Most modern tokenisers split text into subword pieces. In 2015 Rico Sennrich, Barry Haddow and Alexandra Birch at the University of Edinburgh showed that breaking rare words into smaller units let translation systems cope with words they had never seen, using an adapted version of byte pair encoding (BPE)2. BPE starts from single characters and repeatedly merges the most frequent pair into a new symbol2. Common chunks become single tokens, so "encoding" will often split into "encod" and "ing"1. On average, a token covers about four bytes of text1.
Tokenisers differ between models. Meta's Llama 3 uses a vocabulary of 128,000 tokens, and Meta says its new tokeniser raised English compression from 3.17 to 3.94 characters per token3. Google's SentencePiece trains directly from raw sentences, which makes it language independent4.
Tokenisation explains some odd behaviour. GPT-3's authors noted that character-level puzzles are hard for the model because its tokens cover about 0.7 words each5. It also affects cost. A 2023 study found the tokeniser used by ChatGPT and GPT-4 needs about 1.6 times as many tokens for Italian as for English, 3 times for Arabic, and up to 15 times for Shan, a language of Myanmar6. The authors argue this makes commercial AI services dearer and slower for some languages6.
Example
Run the word "encoding" through a BPE tokeniser such as tiktoken and it will often come back as two tokens, "encod" and "ing"1. At GPT-3's average of about 0.7 words per token, a 1,000-word essay is roughly 1,400 tokens5.
Why it matters
Tokens are the unit AI companies bill by and the unit context windows are measured in. The tokeniser therefore decides how much you pay and how much text fits.
Questions people ask
What is the difference between a token and a tokeniser?
A token is one chunk of text, such as a word or part of a word. The tokeniser is the tool that does the splitting and maps each chunk to a number the model can use1.
Why do AI models struggle to count letters in a word?
Because they never see the letters. They see tokens that often cover several characters, about 0.7 words each for GPT-3, so spelling tasks require the model to pull apart pieces it was never shown separately5.
Related terms
Tools that use it
- ChatGPTOpenAI's general chatbot, the most used AI app in the UK and worldwide.
- Meta Llama modelsMeta's family of downloadable large language models, from Llama 2 to the multimodal Llama 4
Sources
- OpenAI, tiktoken README ("What is BPE anyway?"), GitHub
- Sennrich, Haddow and Birch (2015), Neural Machine Translation of Rare Words with Subword Units, arXiv:1508.07909
- Grattafiori et al. (2024), The Llama 3 Herd of Models, arXiv:2407.21783
- Kudo and Richardson (2018), SentencePiece: A simple and language independent subword tokenizer and detokenizer, arXiv:1808.06226
- Brown et al. (2020), Language Models are Few-Shot Learners, arXiv:2005.14165
- Petrov, La Malfa, Torr and Bibi (2023), Language Model Tokenizers Introduce Unfairness Between Languages, arXiv:2305.15425