Quick takeaways
- Evaluate retrieval and generation separately before you combine them.
- Automated metrics like RAGAS and ARES save time, but human review catches what models miss.
- Start with ten to twenty real questions and known-good answers, then expand.
- A RAG system without an eval loop will silently degrade as sources and queries change.
How to evaluate RAG systems
Umar Jamil covers evals and metrics that apply directly to measuring retrieval and answer quality.
Why evaluation comes first
A RAG system can look convincing while producing wrong answers. The model may sound confident, cite a source, and still miss the point. Evaluation is how you catch that before users do.
Good RAG evaluation splits the problem in two: did the system retrieve the right context, and did it generate a faithful, useful answer from that context? Both can fail independently, so you need metrics for each.
Retrieval metrics
Before judging the answer, check whether the retrieved passages could have supported a correct answer. Build a small test set of questions with gold-standard chunks, then score retrieval.
| Metric | What it measures | When to use it |
|---|---|---|
| Recall@k | Whether relevant chunks appear in the top k results | When missing a source is costly, such as legal or medical queries. |
| Precision@k | How many of the top k results are relevant | When you want to keep context clean and avoid noise. |
| Mean Reciprocal Rank (MRR) | How close the first relevant chunk is to the top | When answer quality depends on getting the best source first. |
| Normalized Discounted Cumulative Gain (NDCG) | Ranking quality across multiple relevance levels | When some sources are much better than others. |
Retrieval is the ceiling for answer quality. If the right source is not in the context, the generation stage cannot recover.
Answer relevance
Answer relevance asks whether the generated response addresses the user question. A faithful but off-topic answer is still unhelpful. Score relevance by comparing the question and answer, either with an LLM judge or an embedding similarity score.
- Embedding relevance: Encode the question and answer, then check cosine similarity. Fast but shallow.
- LLM-as-judge: Ask a separate model to rate whether the answer addresses the question on a 1-5 scale. More flexible but adds cost.
- Human rating: Have reviewers score whether the answer solves the original intent. Slowest but most reliable.
Faithfulness and hallucination
Faithfulness measures whether the answer is supported by the retrieved context, not by outside knowledge or model invention. This is the main defense against RAG hallucinations.
Track faithfulness with NLI-style checks, citation verification, or human annotation. A system that refuses well is often better than one that guesses.
RAGAS
RAGAS is an open-source framework that scores RAG pipelines without ground-truth answers. It uses LLM judges to compute metrics such as faithfulness, answer relevance, context precision, and context recall.
- Faithfulness: Are claims in the answer supported by the context?
- Answer relevancy: Does the answer match the question?
- Context precision: How many retrieved items were useful?
- Context recall: Did retrieval cover everything needed for a correct answer?
RAGAS is useful for quick iteration, but treat it as a signal, not ground truth. The same model used for generation and evaluation can share biases.
ARES
ARES takes a different approach. It trains lightweight judges on synthetic data to predict human preferences for context relevance, answer faithfulness, and answer relevance. Once trained, those judges can score new outputs cheaply.
ARES works well when you have enough examples to train a preference model and want to avoid calling a large LLM for every evaluation. The trade-off is setup effort: you need curated human judgments to fine-tune the judge.
Custom evaluations
Benchmark frameworks are starting points. Most production RAG systems need domain-specific checks. Examples include:
- Citation accuracy: Does each sentence cite a source, and is the citation correct?
- Format compliance: Does the answer follow a required structure, such as a table, list, or policy summary?
- Entity match: Are named entities, product codes, and dates consistent with the source?
- Tone and style: Does the answer match the expected voice for the audience?
Build these as pass-fail checks or rubric scores. Automate what you can, and send edge cases to human review.
Human review
Automated metrics are proxies. Human reviewers still catch nuance, ambiguity, and real-world harm. Design review around clear rubrics, not gut feelings.
- Define what a good answer looks like for each question type.
- Use binary or Likert-scale ratings instead of open comments alone.
- Measure inter-annotator agreement to make sure the rubric is clear.
- Compare human scores against automated scores to find metric blind spots.
Building the eval loop
A one-time evaluation is not enough. Sources change, models get updated, and user questions drift. Set up a lightweight loop:
- Maintain a golden test set of real questions with expected answers.
- Run retrieval and generation metrics on every pipeline change.
- Sample production queries weekly for human review.
- Track scores over time and alert when they drop.
- Use failures to prioritize chunking, metadata, or prompt improvements.
Without AI vs. with AI
| Task | Without AI | With AI |
|---|---|---|
| Retrieval scoring | Teams assume retrieval works if the final answer looks reasonable. | Recall, precision, and MRR are measured on a gold-standard test set of real questions. |
| Answer faithfulness | Answers are accepted at face value without checking source support. | Faithfulness metrics verify that every claim traces back to retrieved context. |
| Metric selection | A single generic score drives all RAG decisions. | Multiple metrics cover retrieval, relevance, faithfulness, and citation quality. |
| Human review | Reviewers scan outputs informally and disagree on quality. | Human reviewers use a shared rubric and measure inter-annotator agreement. |
| Continuous improvement | RAG is deployed once and never re-evaluated. | An eval loop reruns tests on pipeline changes and samples production queries. |
FAQ
Can RAGAS replace human evaluation?
No. RAGAS is a fast signal, but human review is still needed for edge cases, nuance, and safety.
Which retrieval metric is most important?
It depends on the use case. Recall matters when missing a source is costly. MRR matters when the top result drives answer quality.
How many test questions do I need?
Start with ten to twenty high-quality, representative questions. Expand once you see patterns in failures.
What is the difference between faithfulness and relevance?
Faithfulness asks whether the answer is supported by the retrieved context. Relevance asks whether the answer addresses the user question.
Should I evaluate the retriever and generator separately?
Yes. If you only evaluate the final answer, you cannot tell whether retrieval or generation caused the failure.
What is a golden test set?
A small set of real questions with known-good answers and source chunks. It becomes the baseline for measuring changes.
How is context precision different from recall?
Precision measures how many retrieved chunks were useful. Recall measures whether the relevant chunks were found at all.
Can I use RAGAS for production monitoring?
RAGAS is useful for iteration signals, but pair it with human review and domain-specific checks for production decisions.
What is an NLI check for faithfulness?
Natural language inference checks whether a hypothesis, the answer, is entailed by the premise, the retrieved context.
How often should I sample production queries for review?
Weekly is a good starting point. Increase frequency after major pipeline changes or when failure rates rise.