LLM evals
LLM evals are the tests that measure whether an AI feature actually works — checking the model's outputs against expected results, quality criteria, or human judgement. They are to AI products what unit tests are to ordinary software: the way you know a change helped instead of guessing.
Because LLM output is non-deterministic and a tweak to a prompt can fix one case while breaking five others, you cannot ship reliably on vibes. Evals turn 'it seems better' into a number. They range from exact-match checks on structured output, to graded scoring by another model (LLM-as-judge), to human review on a fixed test set. Run before and after every change, they tell you whether you actually improved and catch regressions before users do.
When building a product, evals are the difference between a demo and something you can trust in production. The discipline is to build a representative test set early — including the cases that already failed — and to measure each part separately: retrieval quality, final-answer quality, format adherence, safety. We treat evals as core engineering, not an afterthought, because without them every prompt change is a gamble and 'it works on my example' is not a standard we ship on.
// faq
Frequently asked questions
- How do I evaluate an LLM feature without a labelled dataset?
- Start small and grow it. Collect a handful of real or realistic cases with the answers you'd accept, and add every failure you find to the set. From there you can use exact checks for structured output, an LLM-as-judge for open-ended quality, and periodic human review. A modest, honest test set beats no evals and gets better every time something breaks.
- Why can't I just test AI features by trying a few prompts?
- Because LLMs are non-deterministic and changes have side effects — a fix for one input often silently breaks others you didn't check. Trying a few prompts tells you nothing about the cases you didn't try. A repeatable eval set run on every change is the only way to know a tweak helped overall rather than just on the example in front of you.