What problem does RAG solve?
RAG solves the problem of an AI model not knowing your organisation's documents, or knowing only an out-of-date version of the world. Instead of retraining the model, you hand it the right passages at the moment it answers.
Picture a fictional 50-person tour operator, Tidewater Travel. Its booking team keeps asking the same questions about a 90-page terms and procedures handbook. A team lead, Leon, tried a general chatbot: 'Can a customer change the lead passenger name 10 days before departure, and is there a fee?' The chatbot gave a confident, generic answer about airline rules and a fee that Tidewater does not charge. The real answer is in section 4.2 of the handbook, which the chatbot has never seen.
How does retrieval-augmented generation work?
Retrieval-augmented generation works in three stages: retrieve the passages most relevant to the question, add them to the prompt, then let the model generate an answer from them. Anthropic's glossary describes the same idea: relevant information is retrieved from a knowledge base at the time of the query and passed to the model with the original question.
The term comes from a 2020 research paper by Patrick Lewis and colleagues, which combined a language model with a searchable index of documents and found the answers more specific and factual than the model alone.
1. Prepare the documents
The handbook is split into chunks, such as one section or a few paragraphs each, so that a search can return a focused passage rather than the whole book.
2. Index them for meaning
Each chunk is turned into an embedding, a list of numbers that represents its meaning, and stored in a vector database. AWS describes this step as converting data into numerical representations so it can be matched to questions.
3. Retrieve
The question is converted the same way and matched against the index to find the closest chunks. Many systems also use ordinary keyword search alongside it.
4. Augment the prompt
The top chunks are placed in the prompt with an instruction such as 'answer only from these passages and cite the section'.
5. Generate
The model writes the answer from the supplied text, ideally quoting or citing the passages it used.
What does an answer with and without RAG look like?
Without RAG: 'Most travel companies allow name changes for a fee of around 50 to 100 per person, subject to airline policy.' It sounds helpful, but it is not Tidewater's policy, and nobody can check where it came from.
With RAG: the system retrieves section 4.2 and section 4.5 of the handbook, and the model answers: 'Yes. Section 4.2 allows a lead passenger name change up to 7 days before departure at no charge. Section 4.5 says flight-inclusive packages may carry the airline's own change fee, which the booking system shows. Source: Handbook sections 4.2 and 4.5.' Leon can open both sections and confirm the answer in seconds. If the handbook changes, the answer changes as soon as the new version is indexed.
When should you use RAG, and when not?
Use RAG when answers must come from a specific, changing set of documents and people need to see the source. It suits policy handbooks, product documentation, support knowledge bases and contract libraries. Anthropic's glossary notes it is particularly useful for up-to-date information, domain-specific knowledge and explicit citation of sources.
RAG is not the answer when the documents are small enough to paste straight into the prompt, when the task is about style or format rather than facts, or when the source documents themselves are unreliable. It also does not replace permissions: if a passage is retrieved, the model can repeat it, so access controls must apply before retrieval.
How does retrieval-augmented generation go wrong?
RAG usually goes wrong at retrieval rather than generation: if the wrong passage is found, the model writes a confident answer from it. Anthropic notes that the effectiveness of RAG depends on the quality and relevance of the knowledge base and what is retrieved.
Wrong or partial chunk
The passage with the exception sits in a different chunk from the rule, so the answer misses it.
Stale or duplicate versions
Last year's handbook is still indexed beside this year's, and the old one is retrieved.
Model drifts from the text
The model blends retrieved text with its general knowledge. Instructions to answer only from the sources, and visible citations, make this easier to spot.
No good answer exists
The system should say the documents do not cover the question rather than guess.
Where does RAG fit in the Tosheo.ai catalogue?
Retrieval-augmented generation sits in the Building AI Apps category, in the build AI apps course. This lesson is the concept version for anyone who works with AI assistants built on company documents. The Code track after sign-up builds a small RAG app hands-on, from splitting documents and creating embeddings to retrieval and cited answers, with rubric feedback on each step.
Sources and scope
Anthropic Claude documentation, glossary entry for RAG (checked 19 October 2026) ↗
Anthropic Claude documentation, citations feature (checked 19 October 2026) ↗