An LLM evaluation harness is a fixed set of inputs with known-acceptable outputs, run automatically whenever the model, the prompt, the retrieval index or the tool definitions change. It turns "the answers seem worse since Tuesday" into a number that moved on a specific commit. Without one, the deployment is governed by whoever last tried it by hand.
Why demos are not evidence
Every LLM feature is signed off on a demo. The demo works, because the person running it knows which questions work. Production then asks the questions nobody rehearsed, and the failure arrives as a support ticket weeks later with no way to trace which change caused it.
The gap is not model quality. It is that nothing in the pipeline can answer a simple question: is this version better or worse than the last one?
The set is the hard part
A hundred well-chosen cases beat a thousand generated from a template. What makes them well-chosen:
| Case type | What it protects |
|---|---|
| Ordinary questions, in the words users really use | The common path, phrased the way it arrives, not the way a PM writes it. |
| Ambiguous questions with more than one defensible answer | That the system asks instead of guessing. |
| Questions the system should refuse | Scope. A confident answer here is worse than no answer. |
| Questions whose answer is in a restricted document | Leakage. The correct behaviour is to not know. |
| Every production failure, added permanently | Regression on the exact thing that already went wrong once. |
The last row is the one that compounds. A harness that only contains cases someone imagined up front will keep passing while production keeps failing.
Score properties, not strings
Exact-match scoring dies on the first paraphrase, and similarity scoring rewards answers that sound right. What survives is asserting properties:
- Does the answer cite a source that genuinely contains the claim?
- Does it stay inside the scope the system is permitted to answer on?
- Does it refuse where refusal is correct?
- Are the numbers in the answer present in the retrieved context?
Each of those is checkable without a human reading every output, and each maps to a failure that actually costs something.
Retrieval and generation are two measurements
Measure retrieval first: was the passage that contains the answer in the returned set, and where in the ranking? Then measure generation given that the correct passage was supplied.
Combined, a falling score tells you something broke. Separated, it tells you which half — and those have completely different fixes. Chunking and embedding strategy on one side, prompt and model on the other.
Where it runs
- On every change that can move behaviour. Model version, prompt text, retrieval index, chunk size, tool definitions. All of them.
- Pinned versions. Record which model and prompt produced each result, or a regression cannot be attributed to anything.
- A threshold that blocks. A score nobody gates on is a dashboard, not a control.
- A human sample. A small, regular read of real outputs catches the class of problem no assertion was written for.
The documentation side
For systems in scope of the EU AI Act, evaluation is not only engineering hygiene — testing and its results are part of what has to be documented. Building the harness for engineering reasons produces most of that evidence as a by-product, which is a considerably cheaper path than assembling it afterwards for an assessment.