LLM Evals & Judge Calibration
How do you know a change to an AI feature actually made it better, before your users find out the hard way? That question is what evals answer. An eval, short for evaluation, is a test suite for an AI feature: you take a set of real conversations your product had, mark each one good or bad yourself, and score every proposed change against that set the way a normal test suite scores a refactor. Changed the prompt, swapped the model, adjusted a cutoff: re-run the set, compare the score, and a change that makes it worse does not ship. Teams reach for this the moment an AI feature is live and being edited, because AI features can quietly get worse in ways nobody notices for weeks: the answers stay fluent and the tone stays right while the correct-answer rate slides. Interviewers open with this topic for a simple reason: the most common answer to how-do-you-know is "I tested it manually", and that answer tells them you have never been the one who had to explain a regression to a team. Measuring well has its own traps, and the three challenges below make them concrete: how closely an AI grader must match human marks before you trust it, the difference between two success-per-attempt numbers that look interchangeable and are not, and a check for the eval set quietly containing questions the model already knows the answers to.
Key Concepts
- 1A golden set is the definition of better: a saved set of traces (a trace is the record of one full conversation or task the product actually handled), each labeled by a human, that every prompt, model, or threshold change must beat before it ships
- 2Read before you build: error analysis on 20 to 50 real traces sorts their failures into buckets such as made-up facts, missed retrieval, and wrong tool use, and the buckets pick the metrics worth tracking
- 3An LLM judge is a model asked to grade other model outputs. Since it is a model task, it can be wrong in its own way, so the judge itself gets an eval: agreement with human labels is the gate, and the pattern of its disagreements matters more than the rate
- 4Judge biases run in one direction: longer answers graded higher, particular answer positions favored, its own style graded kindly. A one-directional bias quietly upgrades failing answers, and a forgiving tolerance is exactly what lets it hide
- 5pass@k and pass^k answer different questions about the same list of attempts: the chance that at least one of k tries succeeds versus the chance that all k succeed, and users experience the second
Grade the traces yourself
The judge calibration walkthrough
If you have ever had your work graded, you know the setup: a verdict arrives, and whether you agree with it depends entirely on who did the grading. When an AI product runs hundreds of tasks a day, no team can hand-read every run, so the common pattern is an LLM judge: a second model that reads a trace (the written transcript of one run - what was asked, what the agent did, what it answered) and calls it pass or fail. The judge is a model too, with its own blind spots, so calibration is the discipline of counting how often its verdicts agree with a human's (agreement here means: same pass or fail call on the same trace) before anyone lets it grade alone.
Ten agent traces below, from one fictional billing platform. You call each one pass or fail - and you say how confident you are - before either reference grader speaks. Then you meet the two of them: the human reference label (the grade a careful human reviewer gave the same trace), and a mock LLM judge with opinions of its own. The interesting part is not the score - it is the moments where your call and the judge's call agree while the human reference does not, because that is exactly what a calibration table measures before anyone lets a judge grade alone.
Support copilot, Harborlight billing - the refund question.
Your grade
Confidence