"It looks good" isn't a test

Most LLM features are tested by someone trying a few questions and deciding the answers look fine. Then a prompt gets tweaked, a model gets upgraded, and nobody knows whether things got better or worse until a customer notices.

An evaluation fixes that. It's a fixed set of test cases with known good answers, run through the system the same way every time, so a change shows up as numbers you can compare.

Pick the grader for the kind of answer

Not every answer can be checked the same way. When there's one right answer, a fact, a number, or a JSON object with the right fields, check it exactly: the gauge in the figure reads zero when the answer matches. When the answer is open, like a summary or a reply in the right tone, grade it against a rubric, and decide in advance what counts as a pass. In the figure that's 4 out of 5.

Rubric grading is often done by another model. That works, but only if a person checks a sample of its grades. In the figure, a lamp lights on every fifth card and a person confirms the grade.

Rerun on every change, and read the categories

Prompt v1 passes 15 of 20 cards, 75%. Pull the lever to v2 and the same 20 cards run again: 16 of 20, 80%. On the overall number, v2 is better and you'd ship it.

Look at the categories and the story changes. Facts went from 2 of 5 to 5 of 5, but JSON output fell from 5 of 5 to 3 of 5. If another system parses that JSON, v2 just broke it. That's exactly what an overall score hides and a per-category board shows.

How I set up evaluations

  • Start with 20 to 50 real cases from the actual task, including the awkward ones, before building anything clever.
  • Group them by category, and track pass rate per category, not just one number.
  • Use exact checks wherever an answer can be checked exactly; use a rubric only where it can't.
  • Have a person review a sample of model-graded results, especially early on.
  • Run the set on every prompt, model, or retrieval change, and keep the results history.
  • Add every real failure from production as a new test case, so it can't come back unnoticed.