Quick takeaways
- A RAG stack has six layers: source documents, chunking, embeddings, vector storage, retrieval, and generation.
- Retrieval quality matters more than model choice if your sources are good.
- Start with one knowledge base, standard chunking, and a small evaluation set.
RAG stack overview
Fireship demos the core pieces of a RAG stack and how they fit together.
RAG stack overview
A RAG stack is the set of components that turn raw documents into useful, source-backed answers. The stack is usually a pipeline: documents are loaded, split into chunks, converted into vector embeddings, stored in a vector database, retrieved for a query, and passed to a language model for generation.
Each layer affects the others. A bad chunking strategy hurts retrieval even with the best embedding model. A weak retrieval step wastes a strong generation model. Choose each layer for the questions your users actually ask.
Chunking
Chunking decides how much context each passage contains. A chunk should be large enough to keep a complete idea and small enough to retrieve precisely.
Most teams start with a few hundred tokens per chunk, then adjust based on retrieval results. See RAG guide for a deeper discussion of chunking trade-offs.
Embedding models
Embedding models convert text into dense vectors so similar passages sit close together in vector space. Model choice affects retrieval accuracy, cost, and latency.
| Model type | Best for | Notes |
|---|---|---|
| API embeddings | Fast setup, high quality | OpenAI and Cohere are common choices. |
| Open embeddings | Self-hosted or cost control | BGE, E5, and sentence-transformers are widely used. |
| Multimodal embeddings | Images, audio, or mixed content | Useful when the knowledge base is not only text. |
For a deeper guide, read embedding models guide.
Vector databases
Vector databases store embeddings and support fast approximate nearest neighbor search. The right choice depends on hosting model, scale, filtering needs, and team skills.
Hosted options like Pinecone reduce infrastructure work. Self-hosted options like Weaviate, Chroma, pgvector, and Milvus offer more control. For a full comparison, read vector database comparison.
Frameworks
Frameworks glue the layers together. They handle document loading, chunking, embedding, retrieval, and prompt assembly.
- LlamaIndex: Strong for document-centric RAG with many connectors and index types.
- LangChain: Broad ecosystem with chains, agents, and tool integrations.
- Haystack: Good for production retrieval pipelines and evaluation.
- Custom pipelines: Best when you need tight control over each layer.
Pick the framework that matches your team's existing language and deployment preferences. Avoid over-engineering the first version.
Retrieval
Retrieval finds the most relevant chunks for a query. Basic retrieval uses vector similarity. Advanced systems combine keyword search, sparse vectors, and re-ranking.
Read hybrid search guide for practical implementation advice.
Generation
The generation model receives the query and retrieved chunks, then produces the answer. Model choice depends on answer style, reasoning needs, and cost.
- General assistants: GPT-4o, Claude Sonnet, and Gemini Pro handle most RAG answers well.
- Long context: Useful when retrieved chunks are large or numerous.
- Local models: Useful for privacy or cost at scale.
Keep the prompt focused. Include only the retrieved context, the question, and a clear instruction to answer from the context or say it is not found. See AI models comparison for guidance on choosing a generation model.
Evaluation
Evaluate retrieval and generation separately before judging the whole system. Start with a small set of real questions and mark the ideal source passages.
Without AI vs. with AI
| Task | Without AI | With AI |
|---|---|---|
| Architecture | RAG is built as one monolithic script with no clear boundaries. | The stack has distinct layers: documents, chunking, embeddings, storage, retrieval, and generation. |
| Tool selection | Teams adopt the most hyped framework or database without evaluating fit. | Each tool is chosen based on team skills, scale, and query patterns. |
| Chunking | Documents are split with a one-size-fits-all strategy. | Chunking is tuned per document type and validated with retrieval metrics. |
| Retrieval | Only vector search is used, missing exact-term queries. | Hybrid search, metadata filters, and re-ranking are added based on query analysis. |
| Observability | Failures are debugged by guessing which layer broke. | Each layer logs inputs, outputs, and scores so regressions are traceable. |
FAQ
Do I need every layer in the RAG stack?
No. Some teams start with simple keyword search and a small language model. Add layers as retrieval quality and scale demand it.
Which layer should I optimize first?
Start with chunking and retrieval. A better generation model cannot fix bad chunks or missed sources.
Can I use a regular database instead of a vector database?
Yes, if you use full-text search or keyword retrieval. For semantic search at scale, a vector database or pgvector extension is usually easier.
Should I self-host the embedding model?
Self-host when privacy, latency, or token cost matters. APIs are faster to set up and often easier to maintain.
How do I know which chunk size is right?
Test it. Pick ten real questions, try a few chunk sizes, and measure whether the answer is contained in the retrieved passages.
Is RAG stack different from RAG architecture?
They overlap. RAG stack usually refers to the specific tools and models chosen; RAG architecture refers to the broader system design including data flow and evaluation.
What is the cheapest way to start a RAG stack?
Use existing files, a local embedding model or API trial, and an in-memory vector store. Upgrade pieces as quality and scale demand.
Which layer causes the most RAG failures?
Retrieval. If the right chunks are not found, the generation layer cannot recover.
Do I need a framework like LangChain or LlamaIndex?
Not for a prototype. Frameworks help with connectors and patterns, but custom code gives more control at scale.
How do I future-proof my RAG stack?
Keep source documents, embeddings, and evaluation data portable. Avoid vendor-specific formats for your knowledge base.