Evaluation is the fastest way to tell a shipped AI engineer from someone who has built demos. Demos are judged by whether the output looks good. Systems are judged by whether you can prove a change helped — and if you cannot, every future change is a coin flip.
Interviewers know this, which is why "how would you evaluate it?" has become the standard follow-up to any AI system design answer.
Start by splitting the system
The most common weak answer treats an LLM application as one thing to score. Retrieval-augmented generation has at least two failure modes that require different fixes, and a single end-to-end number tells you something is wrong without telling you which half to touch.
- Retrieval fails when the right context never arrives. Measure recall@k against a labelled set of query-to-document pairs. If the correct chunk is not in the top k, no amount of prompt engineering rescues the answer.
- Generation fails when the model ignores or contradicts the context it was given. Measure groundedness (is every claim supported by the retrieved material) and answer relevance (does it address the question asked).
Only after those two do you look at end-to-end quality. Saying this out loud is often enough to answer the question well, because it demonstrates you have debugged one of these systems rather than only built one.
Choosing a metric that supports a decision
Metrics are worth naming only alongside the decision they inform.
Where there is a right answer — classification, extraction, routing, SQL generation — use deterministic metrics. Exact match, F1, execution accuracy for generated queries. These are cheap, stable, and run in CI.
Where output is open-ended, string-overlap metrics like BLEU and ROUGE correlate poorly with quality; they were built for translation and summarisation benchmarks, not for assistant responses. The practical options are:
- Rubric-based grading by a stronger model, scoring against explicit criteria.
- Pairwise comparison between two system versions, which is more reliable than absolute scoring because relative judgements are easier.
- Human review on a sample, which is the ground truth everything else is calibrated against.
Where cost and latency matter, evaluate those too. A change that improves quality by two points and triples time to first token is not obviously an improvement, and treating quality as the only axis is a common blind spot.
The LLM-as-judge question
Expect this one, because it is where hype meets practice.
Using a model to grade model output is the only affordable option at volume, and it has documented biases:
- Position bias — the option presented first tends to win.
- Self-preference — a model rates its own generations higher.
- Verbosity bias — longer answers score better regardless of content.
The mitigations are what the interviewer is listening for: randomise presentation order and average across both orderings; prefer pairwise comparison to absolute scores; use a stronger model than the one under test; and — the one most candidates miss — calibrate the judge against human labels on a sample so you know its agreement rate before you trust it. A judge with 70% agreement with humans is still useful, but only if you know it is 70%.
Building the regression gate
An evaluation that runs only when someone remembers is not an evaluation.
Keep a golden set — a frozen collection of inputs with known-good outcomes — that runs in CI on every prompt, model, retrieval or chunking change, with thresholds that fail the build. Two properties matter more than size:
It grows from real failures. Every production bug becomes a permanent test case. A set written once from imagination covers what you already thought of, which is precisely the set of things that were not going to break.
Part of it is held out. If you tune against the whole set, you optimise the metric rather than the system. Keep a slice you never look at during development.
A few hundred well-chosen examples covering distinct failure modes beats thousands of scraped ones. Coverage of failure modes is the goal; volume is not.
What production adds that offline cannot
Offline sets never cover the distribution of real traffic, so pair them with online signals: thumbs-down rate, escalation to a human, conversation abandonment, retry rate. Route a percentage of traffic to the new version and compare.
The subtlety worth mentioning: user feedback is heavily biased toward the extremes, and absence of complaints is not evidence of quality. Sampled human review of ordinary traffic catches the quiet mediocrity that no user bothers to report.
What separates a strong answer
Weak:
"I'd use BLEU and ROUGE scores, plus human evaluation."
Strong:
"Retrieval and generation fail differently, so I'd measure them separately — recall@k on a labelled query set for retrieval, groundedness for generation. For open-ended output I'd use pairwise LLM grading with randomised order, calibrated against a couple of hundred human labels so I know the judge's agreement rate. That runs as a frozen golden set in CI with a threshold, and it grows from production failures. Online, I'd watch thumbs-down and escalation rate on a canary before a full rollout."
The second answer contains a diagnosis path, a stated bias mitigation, a regression gate and a production signal. That is a complete evaluation story in about forty seconds.
Practice these
- Design a RAG System — where the evaluation loop is part of the design, not an afterthought
- Design an LLM Inference Platform — latency and cost as first-class metrics alongside quality
- Compute Perplexity of a Language Model — the intrinsic metric, and why it is not a product metric
- Reciprocal Rank Fusion for Hybrid Retrieval — a retrieval change you would need to evaluate
Related concepts: LLM evaluation, retrieval-augmented generation, chunking, embeddings.
Frequently asked questions
How large should an evaluation set be? Large enough to detect the effect size you care about, which is usually a few hundred well-chosen examples rather than thousands of scraped ones. Coverage of distinct failure modes matters more than volume.
Is perplexity a useful evaluation metric? For comparing base models on a corpus, yes. As a product metric, no — a model can improve perplexity while getting worse at the task users care about, because perplexity measures how well it predicts text, not whether the answer is correct or useful.
How do you evaluate a system with no ground truth? Use reference-free metrics that check internal consistency — groundedness against the retrieved context, self-consistency across samples — and pairwise comparison between versions, which needs no absolute truth. Then sample for human review.
How often should the golden set be updated? Continuously, by appending production failures, but the existing entries should stay frozen. If you rewrite old cases you lose comparability across time, which is the entire point of keeping the set.
Does evaluation belong in CI or in a separate pipeline? The cheap deterministic slice belongs in CI so a bad prompt never merges. The expensive graded slice usually runs nightly or pre-release, because paying for a few hundred judge calls on every commit is rarely worth it.