Ask a general-purpose chatbot about your company’s leave policy, last quarter’s support tickets or the clauses in a vendor contract, and it will either refuse or, worse, invent a confident answer. The model was never trained on your documents. It simply doesn’t know them.
Retrieval-augmented generation (RAG) is the most practical fix for that problem. Instead of hoping the model “remembers” the right facts, you fetch the relevant facts first and hand them to the model along with the question. The model then writes an answer grounded in that material.
It sounds simple, and the core idea is. Getting it to work reliably is where most teams spend their time.
The one-paragraph version
A RAG system has two halves. The retrieval half searches a knowledge base (PDFs, wiki pages, tickets, database rows) and pulls out the passages most likely to contain the answer. The generation half sends those passages, plus the user’s question, to a large language model with an instruction like “Answer using only the context below. If the answer isn’t there, say so.” The model’s job shifts from recalling facts to reading and summarising them.
Why not just fine-tune the model?
This is the first question most teams ask. Fine-tuning changes a model’s weights by training it on more examples. It’s useful, but it solves a different problem.
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Facts that change, private documents, citations | Style, tone, format, specialised behaviour |
| Updating knowledge | Add or edit documents; no retraining | Retrain the model |
| Can show sources | Yes, you know which passages were used | Not directly |
| Hallucination control | Better, because answers are grounded in retrieved text | Limited |
| Cost to start | Lower | Higher (data preparation, training runs) |
A useful rule of thumb: fine-tune for how the model should behave; use RAG for what it should know. Many production systems use both.
How a RAG pipeline works, step by step
1. Ingest and clean your documents
Collect the source material and convert it to clean text. This step is unglamorous and decisive. Scanned PDFs need OCR. Tables often break during extraction. Headers, footers and page numbers repeated on every page add noise that later confuses search.
2. Split documents into chunks
Models and search systems work better with passages than with 80-page documents. Each document is split into chunks, typically a few hundred tokens long, often with a small overlap so a sentence isn’t cut in half at a boundary.
Chunking choices matter more than people expect:
- Too small, and a chunk loses context (“It must be submitted within 30 days.” What must?).
- Too large, and a chunk mixes several topics, so search matches it loosely.
- Structure-aware chunking, splitting on headings, sections or FAQ entries, usually beats fixed-length splitting for policy documents and manuals.
Store useful metadata with each chunk: document title, section heading, date, department, access level. You’ll need it for filtering and citations.
3. Create embeddings
An embedding model converts each chunk into a vector, a long list of numbers that represents its meaning. Chunks about similar ideas end up close together in this vector space, even if they use different words. “Annual leave” and “paid time off” land near each other.
4. Store vectors in an index
The vectors go into a vector database or a search engine with vector support. Options range from dedicated vector databases to extensions for databases you may already run, such as pgvector for PostgreSQL.
5. Retrieve relevant chunks at query time
When a user asks a question, the question is embedded with the same model, and the index returns the chunks whose vectors are closest. Most systems retrieve somewhere between 3 and 20 chunks.
6. Generate the answer
The retrieved chunks are inserted into a prompt along with instructions and the question. A typical prompt says: answer only from the context, cite the source for each claim, and say “I don’t know” when the context doesn’t contain the answer.
Where RAG systems usually fail
Most RAG problems are retrieval problems, not model problems. If the right passage never reaches the model, no model can answer correctly.
Vocabulary mismatch. Users ask “How do I get my money back?” while the policy says “refund procedure.” Embeddings help, but not always. Hybrid search, combining vector search with traditional keyword search (BM25), catches exact terms like product codes, error numbers and names that pure semantic search can miss.
Wrong chunk wins. The top result is about a similar topic but not the right one. A re-ranker, a second model that scores each retrieved chunk against the question more carefully, often improves precision noticeably.
Outdated or conflicting documents. If the 2023 and 2026 versions of a policy are both indexed, the model may quote the old one. Use metadata to prefer the latest version, and remove superseded documents.
Multi-part questions. “Compare the travel allowance for managers and engineers” may need passages from two sections. Query rewriting or splitting the question into sub-questions helps.
Access control ignored. If HR salary bands are in the index, the retrieval layer must filter by the user’s permissions before anything reaches the model. Never rely on the prompt to hide information.
A practical checklist before going live
- Build a test set of 50–100 real questions with known correct answers and sources.
- Measure retrieval separately: for each question, did the correct chunk appear in the top results?
- Measure answer quality: is the answer correct, complete and supported by the cited text?
- Test “no answer” cases: questions your documents don’t cover should get an honest “I don’t know.”
- Log everything: questions, retrieved chunks and answers, so you can debug failures.
- Plan updates: decide how new and changed documents get re-indexed.
Where RAG is genuinely useful
- Internal knowledge assistants for HR policies, IT runbooks and engineering wikis
- Customer support answering from product manuals and past tickets
- Legal and compliance teams searching contracts and regulations, with citations for review
- Developer documentation chat over API references and code examples
- Government and procurement teams querying tender documents and guidelines
What RAG does not fix
RAG reduces hallucinations; it doesn’t eliminate them. A model can still misread a passage, merge two sources incorrectly or over-generalise. For high-stakes uses, show citations prominently and keep a human reviewer in the loop.
It also won’t fix bad source material. If your documentation is outdated, contradictory or incomplete, RAG will faithfully surface those problems. Teams often find that their first RAG project turns into a documentation clean-up project, and that’s not a bad outcome.
The bottom line
RAG works because it plays to a language model’s real strength, reading and summarising, instead of relying on what it memorised during training. The model is rarely the weak link. Clean documents, sensible chunking, hybrid retrieval, permission filters and a proper test set are what separate a demo from a system people trust.
Start with one well-defined knowledge base, measure retrieval honestly, and expand only when the answers hold up.
Leave a Comment