Retrieval-augmented generation
In short
Retrieval-augmented generation (RAG) is a way of making an AI chatbot answer from a set of documents. Before replying, the system searches those documents for relevant passages and pastes them into the prompt, so the answer can draw on them.
- Also called
- RAG, Retrieval augmented generation
- Topic
- Using AI tools
- Level
- Some background helps
- First described
- Lewis and colleagues at Facebook AI Research, UCL and NYU, May 2020, in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"1.
A language model only knows what was in its training data. RAG adds a lookup step. The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, which paired a text generator with a searchable index of Wikipedia1. They found answers became "more specific, diverse and factual" than from the model alone1.
Today's RAG systems usually work in stages, as Anthropic describes. Documents are cut into chunks of a few hundred tokens. An embedding model turns each chunk into a list of numbers that captures its meaning, and these are stored in a vector database. When a question arrives, the closest chunks are found and added to the prompt2. Many systems also run a keyword search, because meaning-based search can miss exact strings such as an error code2.
Retrieval quality matters. Anthropic reported that adding a short explanatory note to each chunk before indexing cut failed retrievals by 49%, and by 67% when combined with a reranking step2.
RAG is not always needed. Anthropic suggests that a knowledge base under about 200,000 tokens, roughly 500 pages, can simply be put into the prompt whole2. And RAG reduces made-up answers rather than ending them: the model can still misread or ignore what it retrieved.
Developers meet it as a ready-made feature. OpenAI's file search tool, for example, lets a model search uploaded files using both meaning-based and keyword search, after they are loaded into a "vector store"3.
Example
A housing association loads its tenancy agreements and repairs policy into a RAG chatbot. A tenant asks "Who fixes a broken boiler?" and the bot finds the repairs clause and answers from it, rather than guessing from general knowledge.
Why it matters
It is the most common way companies make chatbots answer from their own, up-to-date information. When a tool says it answers "from your documents", it is almost always doing some form of RAG.
Questions people ask
What does RAG stand for in AI?
Retrieval-augmented generation. The model retrieves relevant text from a document collection and uses it to generate its answer. The name comes from a 2020 paper by Lewis and colleagues1.
Is RAG better than fine-tuning?
They do different jobs. RAG feeds the model fresh facts at question time without changing it, so documents can be updated instantly. Fine-tuning changes the model itself and suits teaching a style or format. Many products use RAG for facts because it is easier to keep current12.
Does RAG stop hallucinations?
It reduces them but does not stop them. The original RAG paper found answers became more factual1, but a model can still pick the wrong passage or misstate what it says. Checking the cited source is still wise.