Definition
LLM evaluation is the practice of measuring an application's output quality against a fixed dataset and a defined scoring method, so that changes to prompts, models, or retrieval can be compared rather than guessed at. Scoring ranges from exact-match and programmatic checks, through similarity metrics, to model-graded rubrics where another model judges outputs — each progressively more flexible and less reliable.
Evals matter more in language-model systems than tests do in ordinary software because the behavior is non-deterministic and the failure modes are subtle: a prompt edit that fixes one case routinely breaks three others, and nobody notices without a scored suite. The discipline that separates working teams from stuck ones is building a small evaluation set early — often just a few dozen real failure cases — and growing it from production incidents, so that every regression seen once becomes a case that cannot silently return.
How it comes up in interviews
This is repeatedly reported as the topic AI-engineering candidates are least prepared for, which makes it a cheap way to stand out. When asked how you would ship a language-model feature, describe the evaluation loop before the architecture: what the dataset is, who labels it, what the score means, and what score gates a release. Also be honest about model-graded evaluation's weaknesses — position bias, self-preference, and the need to spot-check the judge.
What separates a strong answer
Common answer
Listing metric names — BLEU, ROUGE, perplexity — without saying which decision each one supports.
What lands
Splitting the system before measuring it: "retrieval and generation fail differently, so I measure recall@k on retrieval and groundedness on generation — otherwise a bad answer tells me nothing about which half to fix."
LLM Evaluation (Evals) interview questions
The forms this comes up in, and what a complete answer covers.