Quick takeaways
- Your real task is the only benchmark that matters for your workflow.
- Define the task, build a rubric, then collect inputs before you run any metric.
- Combine automated metrics for scale with human review for judgment.
- Track scores over time so you catch regressions after model updates.
How to evaluate large language models
Umar Jamil walks through LLM evaluation methods, benchmarks, and how to choose the right metrics.
Why run your own evals
Public benchmarks measure general capability, not whether a model fits your prompts, your data, and your quality bar. A model that tops a coding benchmark may still write summaries your team dislikes. Your own evaluation closes that gap.
Internal evals also protect you from benchmark gaming, hidden prompting tricks, and overfitting to stale test sets. They become the source of truth for which model to use and when to switch.
Define the task
Evaluation starts with a clear task definition. "Write better emails" is too vague. "Generate a follow-up email from a meeting transcript, matching our tone and including action items" is something you can score.
- What is the input?
- What should the output look like?
- What are common failure modes?
- Who will review the output?
Build a rubric
A rubric turns subjective quality into repeatable scores. Use dimensions that match your task, and keep the scale simple.
| Dimension | 1 | 3 | 5 |
|---|---|---|---|
| Correctness | Factually wrong | Mostly right, minor errors | Fully accurate |
| Clarity | Hard to follow | Understandable with effort | Clear and well structured |
| Tone | Mismatched | Neutral | Matches brand voice |
| Completeness | Missing key points | Covers main points | Thorough without fluff |
Include examples of good and bad outputs for each level. This reduces disagreement between reviewers.
Create a test set
Your test set should represent real usage. Pull examples from production logs, support tickets, or representative prompts. Include edge cases and known failure modes, not just easy wins.
- Collect 20-50 diverse inputs.
- Write the ideal output or scoring notes for each.
- Label difficulty: easy, typical, or adversarial.
- Keep the test set separate from training or prompt engineering data.
Automated metrics
Automated metrics let you evaluate at scale. Pick metrics that match your task, and do not trust any single number.
Human review
Human reviewers catch what automated metrics miss: tone, subtle inaccuracies, brand risk, and user intent. Make review structured and repeatable.
- Score each example with the rubric, then average scores per model.
- Record disagreements and refine the rubric when reviewers diverge.
- Review a random sample of production outputs, not just test-set outputs.
- Track the cost of review so you know when automation is worth building.
Regression tracking
Model updates can quietly hurt performance. Regression tracking compares new model versions against your baseline on the same test set.
- Record the baseline score for each metric and rubric dimension.
- Re-run the full test set on every candidate model or prompt change.
- Flag any dimension that drops more than your tolerance.
- Investigate failures before shipping the change.
Store scores in a simple spreadsheet, database, or evaluation platform. The important part is comparing apples to apples over time.
Common mistakes
Without AI vs. with AI
| Task | Without AI | With AI |
|---|---|---|
| Model selection | Teams pick models based on leaderboard screenshots and vendor demos. | Teams run their own prompts, rubrics, and test sets to find the best model for the task. |
| Quality scoring | Reviewers argue about whether an output is "good enough" without shared criteria. | A rubric scores correctness, clarity, tone, and completeness so scores are repeatable. |
| Test sets | Prompts are tested ad hoc during implementation and forgotten afterward. | A curated test set represents real inputs, edge cases, and known failure modes. |
| Automation | Every evaluation is run manually in a notebook or chat interface. | Automated metrics and LLM-as-judge run at scale, with human review on samples and disputes. |
| Regression tracking | Model updates are shipped without checking whether quality drops. | Every candidate model is scored against the baseline and flagged if any dimension regresses. |
FAQ
How big should my test set be?
Start with 20-50 representative examples. Expand once you see consistent failure patterns or before making high-stakes model decisions.
Can I evaluate without human reviewers?
For early screening, yes. For final decisions, include at least a sample of human review to catch what metrics miss.
How often should I re-run evaluations?
Run them when you change models, prompts, or workflows. At minimum, review a sample of production outputs weekly.
What if two models tie on my rubric?
Break ties with cost, speed, context window, safety, or ease of integration. The best model is not always the highest-scoring one.
Should I use public benchmarks at all?
Yes, as a rough filter. They help narrow the field, but your own evaluation should make the final call.
What is an LLM-as-judge?
It is using a separate language model to score outputs against a rubric. It is flexible but can share biases with the model being evaluated.
How do I avoid overfitting my test set?
Keep the test set separate from prompt engineering, limit how often you optimize against it, and refresh it with new production examples.
Can automated metrics replace human review?
No. Automated metrics scale screening; human review catches nuance, safety, and real-world intent.
What dimensions should my rubric include?
Common dimensions are correctness, clarity, tone, completeness, format compliance, and safety. Pick the ones that matter for your task.
How do I compare proprietary and open models fairly?
Use identical prompts, temperature, and max tokens; score on the same rubric; and include cost and latency in the comparison.