Retrieval-augmented generation (RAG) is a way to make a language model answer from your own information. Before the model writes, the system searches a knowledge source, such as company documents or a database, pulls the most relevant passages and gives them to the model with the question, so the answer is grounded in those sources and can cite them.
RAG exists because language models on their own only know what was in their training data, cannot see your private documents, and will sometimes produce a confident answer that is wrong. Retrieval fixes the first two problems directly and greatly reduces the third. The term comes from a 2020 research paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which combined a retriever with a generator so the model could draw on an external index and show where its answers came from.
How RAG works
A RAG system has two halves: an indexing pipeline that prepares your content ahead of time, and a query pipeline that runs every time someone asks a question.
Indexing: preparing the knowledge
- Ingest. Connect sources such as SharePoint, Google Drive, Confluence, a ticketing system or a database. Parse PDFs, slides, tables and scanned pages, running OCR where needed.
- Chunk. Split documents into passages small enough to retrieve precisely but large enough to keep their meaning. Chunking by section and heading usually beats fixed-length cuts.
- Embed. Convert each chunk into a vector, a list of numbers that captures its meaning, using an embedding model.
- Store with metadata. Save the vectors in a vector database with metadata such as source, date, owner and access permissions.
Querying: answering a question
- Retrieve. Turn the question into a vector and find the closest chunks. Most production systems combine this semantic search with keyword search (hybrid search) so exact terms such as product codes are not missed.
- Filter. Remove chunks the user is not allowed to see, or that are out of date.
- Rerank. Use a second, more precise model to reorder the candidates and keep only the best.
- Generate. Give the model the question and the selected passages, with instructions to answer only from them and cite each claim.
- Check. Optionally verify that the answer is supported by the passages, and decline if it is not.
What decides RAG answer quality
When a RAG system gives bad answers, the model is rarely the cause. The usual culprits are upstream:
- Retrieval misses. If the right passage never reaches the model, no model can answer correctly. Measure retrieval separately from generation.
- Poor parsing. Tables flattened into nonsense or headers lost from chunks make passages unusable.
- Stale content. Old versions of a policy retrieved alongside new ones produce contradictions. Freshness pipelines and deduplication fix this.
- No permission model. A RAG system that ignores access controls will show people documents they should not see. Permissions have to be enforced at retrieval time.
- No refusal rule. The model should say it does not know when the sources do not support an answer. Without that rule, RAG still produces hallucinations, just better-sounding ones.
RAG vs fine-tuning
RAG gives a model access to knowledge at the moment of the question. Fine-tuning changes the model itself by training it further on examples. They solve different problems.
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Facts, documents and data that change | Style, format and narrow task behavior |
| Updating knowledge | Re-index the documents | Retrain the model |
| Citations | Yes, to the retrieved passage | No |
| Access control | Per user, at retrieval | Not possible inside the model |
Many systems use both. We cover the trade-offs in fine-tuning vs RAG.
Where RAG is used
- Internal knowledge assistants that answer staff questions from policies, wikis and SOPs. See AI knowledge base.
- Customer support bots grounded in help docs and product catalogs.
- Legal, compliance and regulated work where every answer must point to a source.
- Technical and engineering search over code, runbooks and tickets.
- Agents that need company knowledge before they act. Most AI agents include a RAG step.
Krazimo example: legal AI that must cite or refuse
For CaseLogic, a state-aware legal assistant, a confident wrong answer has legal consequences, so retrieval quality is the product. The RAG pipeline pulls more than 500 candidate chunks for a question, then a neural reranker narrows them to about 50 of the most relevant before the model sees anything.
Documents for every case live in one vector store but are isolated by case metadata, so retrieval never crosses into another client’s workspace. Every substantive answer must cite the retrieved chunks, and if the system cannot find grounding for a claim it must refuse. A separate reflection agent then checks the draft for unsupported claims, missing citations and inconsistencies with the retrieved text before the user sees it.
The same principle runs through Krazimo Studio, our platform for grounded AI agents that cite their sources and decline what those sources do not support.
How to evaluate a RAG system
Test the two halves separately, on questions your users actually ask:
- Retrieval: for each test question, did the correct passage appear in the top results?
- Faithfulness: is every claim in the answer supported by a retrieved passage?
- Correctness: does the answer match what an expert would say?
- Refusal: on questions the sources cannot answer, does the system decline?
Build the test set from real user questions, including ones the sources cannot answer, and rerun it every time you change the chunking, the embedding model, the reranker or the prompt. Small changes upstream often move answer quality more than switching models does.
Our LLM evaluation page explains how to build that test set. Our RAG development service includes it from the start, along with ingestion, permissions and monthly tuning.
FAQ
What does RAG stand for in AI?
RAG stands for retrieval-augmented generation. The system retrieves relevant information from a knowledge source and adds it to the model’s input, so the generated answer is based on that information rather than only on what the model learned in training.
Does RAG stop hallucinations?
It reduces them a lot but does not remove them on its own. The model can still misread a passage or fill gaps when retrieval misses. Instructions to answer only from sources, citation requirements, a refusal rule and an answer check close most of the remaining gap.
What is a vector database and do I need one for RAG?
A vector database stores text as numeric vectors so it can find passages with similar meaning to a question. Most RAG systems use one, often alongside keyword search. For small collections a regular database with vector support is usually enough.
Can RAG work with private or sensitive data?
Yes. The documents, index and model can all run inside your own cloud or servers, and permissions can be applied per user at retrieval time. For workloads that cannot leave your infrastructure, see what is a private LLM.
Is RAG better than fine-tuning?
For knowledge that changes or must be cited, yes. For teaching a model a consistent format, tone or narrow skill, fine-tuning can be better. Many production systems use RAG for facts and light fine-tuning for behavior.
How long does it take to build a RAG system?
A working pilot on one set of sources typically takes a few weeks. Connectors, permission mapping, evaluation and tuning for production add time depending on how many systems are involved. On cost, Krazimo’s Clutch profile lists a minimum project size of $10,000+ and rates of $100-149 per hour; scope sets the final price. See AI agent development cost or contact us for an estimate.