Quick takeaways
- The quality of RAG depends on retrieval and evaluation more than the chat UI.
- Start with one knowledge base and real questions.
- If your sources are messy, fix the knowledge first.
RAG fundamentals for teams
IBM Technology covers retrieval, generation, and why RAG is the default pattern for knowledge systems.
RAG pipeline
A practical RAG pipeline stores documents, chunks them, indexes them, retrieves relevant passages, and uses those passages to ground the answer. The model is not replacing knowledge; it is using the right context at answer time.
The simplest version of RAG is: a question comes in, the system finds the most relevant chunks, and the model answers based only on those chunks. More advanced versions re-rank chunks, summarize across sources, and cite passages explicitly.
What to store
- Policies, FAQs, documentation, tickets, transcripts, product specs, and research notes.
- Metadata that helps the system know what is current, approved, or source of truth.
- Examples of good answers so the model knows the style and level of detail to target.
Tool stack
A production RAG system combines document processing, embeddings, a vector store, an orchestration layer, and evaluation. Here are common choices by layer.
| Layer | Popular options | When to use |
|---|---|---|
| Vector database | Pinecone, Weaviate, Chroma, pgvector, Milvus | Semantic search at scale; choose hosted vs self-hosted based on ops capacity. |
| Embeddings | OpenAI, Cohere, BGE, E5, sentence-transformers | Convert chunks into searchable vectors; match model to language and domain. |
| Frameworks | LangChain, LlamaIndex, Haystack, custom | Orchestrate chunking, retrieval, and generation; custom stacks reduce lock-in. |
| Evaluation | RAGAS, ARES, custom rubrics | Measure retrieval and answer quality continuously. |
See the full RAG stack guide, vector database comparison, and embedding models guide for deeper comparisons.
Chunking strategies
Chunking is the most underrated part of RAG. Bad chunks break retrieval even when the documents are good. A good chunk preserves a complete thought and enough context to be useful on its own.
Retrieval evaluation
Before you evaluate the final answer, evaluate whether the right chunks were retrieved. Create a small set of real questions and mark the gold-standard passages. Then measure how often the system finds them.
- Recall: Did the system retrieve the relevant chunks?
- Precision: How many of the retrieved chunks were actually relevant?
- Mean reciprocal rank: How close to the top was the first relevant chunk?
RAG evaluation frameworks
Beyond retrieval metrics, evaluate the end-to-end RAG output. Frameworks like RAGAS and ARES provide automated scores for answer faithfulness, relevance, and context precision. Use them as a signal, not a substitute for human review.
- Faithfulness: Does the answer stick to the retrieved context?
- Answer relevance: Does the answer address the question?
- Context precision/recall: Were the right chunks used?
For a full evaluation framework, see RAG evaluation framework.
Hybrid search and re-ranking
Pure vector search works well for semantic similarity but can miss exact matches like IDs, product names, or rare terms. Hybrid search combines keyword search (BM25) with vector search, then re-ranks the combined results.
- Use keyword search for exact terms and identifiers.
- Use vector search for conceptual similarity and paraphrases.
- Use a re-ranker to put the most relevant results at the top.
Read more in the hybrid search guide.
Source hygiene
RAG cannot fix bad source material. If your documents contradict each other, are out of date, or use inconsistent terminology, the answers will reflect that. Source hygiene means:
- Removing or archiving outdated documents.
- Marking the authoritative version when duplicates exist.
- Adding metadata such as last-updated date, owner, and approval status.
- Running periodic audits on the most-queried topics.
Failure modes
Testing RAG
Start testing with ten to twenty real questions. Do not optimize for synthetic benchmarks until the system handles the questions your users actually ask.
Without AI vs. with AI
| Task | Without AI | With AI |
|---|---|---|
| Retrieval design | Teams pick a vector database by brand recognition without testing on real queries. | Teams benchmark retrieval on real questions and choose chunking, embeddings, and search by measured results. |
| Chunking | Documents are split by fixed size with no overlap or semantic boundaries. | Chunks preserve complete thoughts, include useful metadata, and are tuned using retrieval scores. |
| Source hygiene | Old policies, duplicates, and drafts live in the same index as current truth. | Sources are audited, deduplicated, and tagged with owner, freshness, and authority status. |
| Evaluation | Quality is judged by whether the answer looks plausible. | Teams measure retrieval recall, answer faithfulness, and citation accuracy on a held-out test set. |
| Deployment | A chatbot is launched to all users before failure modes are understood. | RAG rolls out to a small audience with logging, feedback, and a clear escalation path. |
FAQ
Is RAG only for enterprises?
No. Small teams use RAG for docs, support content, sales notes, and knowledge bases.
What breaks RAG most often?
Stale documents, poor chunking, bad retrieval, and weak evaluation.
How big should chunks be?
Big enough to preserve meaning, small enough to retrieve precisely. Most systems use a few hundred tokens per chunk as a starting point.
Do I need a vector database?
Not always. Some use cases work with keyword search, hybrid search, or a managed retrieval service. Choose based on scale and query patterns.
How do I know if my RAG system is improving?
Track retrieval accuracy, answer correctness, user satisfaction, and how often answers cite the right source.
Can RAG handle conflicting sources?
It can surface conflicts, but you need source hygiene and clear rules about which document is authoritative.
How do I choose between dense and hybrid retrieval?
Start with dense retrieval for conceptual queries. Add hybrid search once you see exact IDs, product names, or rare terms being missed.
What is the right team size for a RAG project?
A RAG prototype needs one developer and one content owner. Production systems benefit from a small team with engineering, domain, and operations skills.
Should I build RAG in-house or use a platform?
Build in-house when you need tight control over data and evaluation. Use a platform when speed matters more than customization.
How do I prevent source conflicts in RAG?
Mark an authoritative version for each topic, archive stale documents, and surface conflicting sources instead of hiding them.