LLM evals: calibrating the judge
Ask a chatbot the same question five times and mostly you get a good answer, sometimes a confident mess, and no score anywhere tells you which is which. That absence is why this subject exists. An eval is the industry word for turning that feeling into a number: you keep a fixed set of realistic test questions with the system's answers saved on them, you write your own grade next to every answer, and then each time you change the prompt or swap the model you run the same set again and check whether the grades actually got better. The saved record of one run, question in and answer out plus everything in between, is called a trace. The graded set is often named the golden set, because it is the one fixed definition of "better" the whole team measures against. Nobody changes a prompt by hand and ships it on vibes; they run it against the golden set and quote a number. Here is the part nobody warns you about. When there are thousands of answers and no time to grade them all, a common shortcut is to use another AI model as the grader - people call it a judge. That works, but a judge is just another model making guesses about your product, and the only thing that makes it trustworthy is measuring how often its grades match a human's on the same pile. Ninety percent agreement sounds shippable until you look at where the other ten percent lives. Every mismatch pile I have seen clusters in the same place: the borderline answers, the exact ones your decisions turn on. Agreement on the easy middle cases was flattering noise.
Most people have this backwards
A judge that agrees with your labels 90 percent of the time is not a judge you can trust. The proof lives in the other 10 percent.
The catch
It holds when the disagreement concentrates on borderline traces, because the borderline ones are the cases a product decision turns on, and agreement on the easy middle cases inflates the headline number without buying any protection. The claim breaks where disagreements are uniformly spread noise on a well-labeled axis: then the overall agreement is at least an honest first read, and the right response is more labels rather than suspicion.
Baseline first - what is your honest read?
A colleague shows you an LLM judge scoring every trace automatically plus a handful of their own spot checks that mostly agree, and proposes shipping it as the release gate. What do you ask first?
Make each call, then see the verdict
Predict first
A multi-step checkout handled by an AI assistant: each run succeeds 90 percent of the time in your test setup. A real user session chains three of those runs back to back. Does the share of sessions where all three succeed land near 90 percent, near 73 percent, or near 100 percent?
Four judge setups sitting in front of a release gate. Decide for each whether the gate runs.
The judge agrees with human labels on 94 percent of your labeled traces, and you read the mismatches: they split about evenly between over-grading and under-grading.
The judge scores 88 percent agreement on your labeled set, and every single mismatch is the same move: the judge grades a failing trace one band higher.
The judge was never measured. It writes scores into the dashboard, the metric has looked stable for two weeks, and nobody can say what those scores mean.
The judge is fully calibrated on your labels and scores 97 percent agreement, but its score now triggers an automatic refund, an action nobody in the room can undo.
How it works
The pipeline reads best from the trace outward. A trace, again, is one saved run: the user's request, whatever the system looked up along the way, the answer the model produced, and, if the system calls tools, those calls too. Start with your own hands: read a small pile of traces, twenty to fifty, and write a grade on each one. That pile, frozen so it never gets edited to flatter a change, is the golden set, and it is the definition of better. You cannot hand-grade thousands of answers forever, so past that pile a judge model does the grading - and here is the step people skip: the judge itself gets measured. On the traces you hand-graded, compare the judge's grade against yours and look at two things: how often the two agree exactly, and what every disagreement looks like. Those disagreements are the judge's report card, and they are also where its bias hides. Last comes the gate: a rule that says a prompt or model change ships when the judge score clears a threshold, and the threshold is set by whoever owns the product risk, not by whoever finds a number convenient. Two smaller habits brace the whole thing. Before trusting any scores, check that none of your golden-set questions leaked into the examples the model was trained on, so nothing on the set grades itself. And when you sample the same task repeatedly, keep two success rates: pass@k, the chance of getting a case right within k tries, and pass^k, the chance of succeeding on every leg of one continuous session, because they answer different questions and disagree on purpose.
The thermometer check
The picture is calibrating a thermometer. You do not read the manual and trust the instrument; you stick it in ice water and in boiling water, two points whose truth you already know, and you check the reading. Judge calibration is that move: grade a small pile of traces by hand, and compare the judge reading where you already know the truth. Every working eval setup I have read about does this somewhere, a small pile of hand labels used as ice water.
Where it breaks
The analogy breaks at the direction of the drift. A bad thermometer is usually wrong by roughly the same amount across its whole range, so two check points bound the error everywhere. A judge drifts by neighborhood: it reads excellent on clear examples, and the error concentrates on borderline ones, which is the temperature range where the decisions live. Two agreed-upon check points will not bound that, so the hand-labeled pile has to be drawn where answers are genuinely hard, which is also why a small honest set beats a large one full of easy reads.
Why interviewers lean on it
This subject comes up constantly in interviews because it is the fastest divider between someone who has actually run these checks and someone who has read about them. The phrase that ends a round is a candidate who answered the "how do you know it is better" question with some version of "I tested it manually", which, without a fixed set and a number, means they felt it. A candidate who answers with a number plus the pattern of disagreements changes the subject of the interview. The questions after that stay quantitative on purpose: what was the exact agreement with the hand labels, which direction do the disagreements move, whose grades set the bar, and what did a looser version of the agreement number hide. This is not only an interview game. An unmeasured judge is a real product liability, because a judge with a lenient streak quietly passes the very failures it was installed to catch.
The follow-up chain
Interviewers do not stop at the surface answer. Answer the way you would out loud, or skip - skipping reveals the strong answer and ends the chain.
The interviewer's next question
You say the new prompt is better. Better on what, measured how?
Now answer one you have not seen
A teammate ships an LLM judge to gate prompt changes in CI. It shows 90 percent agreement with their own spot checks, and the team dashboard has been flat for two weeks. You get one afternoon before the gate goes live. What do you bring to the review, and what do you refuse to accept yet?
Expires fast - as of 2026-10-05
The practice of calibrating an LLM judge against human labels to above 90 percent exact agreement, and error analysis on 20 to 50 traces before building any eval infrastructure, come from 2026 interview-loop research: evals appear in 72 percent of the 25 posted job descriptions mined for this track, and a first-class eval round is documented at OpenAI.
pass^k comes from the tau-bench lineage of agent evaluations. The frequently quoted pairing: a 70 percent per-run success rate is about 97 percent pass@k on one axis and about 34 percent pass^k on the other at the relevant k, and agent-eval writeups treat both numbers as mandatory.
As of 2026-10, the manual workflow itself (trace review, bucketing failures, judge calibration rounds) appears in no prep-site curriculum reviewed for this track, and binary pass/fail over Likert grades remains the common interview shorthand for the calibration question.