LLM as a judge evaluation: three biases to correct

If you are evaluating an AI feature at any scale, something is grading the output, and it is probably another model. LLM as a judge evaluation works better than most people expect and fails in three specific, measurable directions — all of which have cheap corrections that almost nobody applies.

Start with the encouraging number, because it is the reason this is worth doing at all.

It agrees with humans more than humans do

The reference work is Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Zheng and colleagues, and its headline finding still frames the field: GPT-4 reached roughly 85% agreement with human annotators, higher than the roughly 81% agreement humans achieved with each other.

That second figure deserves more attention than it gets. Human evaluation is treated as the gold standard, and two careful annotators disagree about one judgement in five. So the question was never whether model grading is perfect. It is whether it is more consistent and vastly cheaper than the alternative, and on those terms it wins comfortably.

The failure modes, catalogued in the same paper, are what you have to control for.

One caveat on the headline figure before leaning on it. That 85% was measured on a particular benchmark of chat-assistant answers with GPT-4 as the judge, not on your domain with your rubric. Agreement on a specialist task — a clinical summary, a legal clause, a pricing explanation — is typically lower, because the disagreements are about domain judgement rather than language quality. The number tells you the approach is viable. It does not tell you your judge is accurate.

Three biases, and what each costs

Position bias. In a pairwise comparison the judge favours whichever response appears first. This is the most dangerous of the three because it is invisible — your results look clean, they are simply partly a function of ordering.

Verbosity bias. Longer answers score higher regardless of actual quality. If you are using judge scores to tune prompts, you will drift toward a more verbose product without ever deciding to, because every iteration that padded the answer scored better.

Self-enhancement bias. A model rates outputs matching its own style more favourably. The paper puts this at around a 10% higher win rate for GPT-4 judging its own answers, and about 25% for Claude-v1. Which means the most common setup — using the same model family to generate and to grade — is the one carrying the largest thumb on the scale.

There is a fourth limitation the paper names that gets less airtime: limited reasoning ability. A judge asked to verify a multi-step calculation, or to decide whether an argument follows, is being asked to do the hard task twice — once to work out the right answer and once to compare. On anything requiring real reasoning to grade, judge scores degrade in ways that do not show up as disagreement so much as confident error, and the correction is to supply the reference answer rather than expect the judge to derive it.

These compound, which is the practical problem. A verbose answer generated by the same model family that is grading it, presented first, benefits from all three at once — and nothing in the output looks unusual. That is what makes model grading feel trustworthy while being systematically tilted: the bias is in the instrument rather than in any individual reading, so no single score looks wrong.

The corrections cost almost nothing

None of these require a different model or a research budget.

Swap and average. Run every pairwise comparison twice with the order reversed and average the result. This removes position bias almost entirely. It doubles your judging cost, which on an eval suite is trivial, and it is the single highest-value change on this list.

Hide the identity. The judge should not know which system produced which answer. Label them A and B, and randomise which is which per comparison.

Instruct explicitly against style. Tell the judge to ignore length, formatting and tone, and to score only whether the answer is correct and complete. Blunt, and it measurably helps.

Use few-shot examples. Showing the judge a handful of graded examples raised GPT-4 scoring consistency from 65.0% to 77.5% in the paper. Consistency is the property you most need from a judge, because an inconsistent grader makes every comparison between two runs meaningless.

Grade with a different model family than you generate with. This follows directly from self-enhancement bias and is the correction teams skip most often, usually because one provider is already wired in.

Worth noting what these corrections have in common: each removes information the judge should not be using. Order, length, style and provenance are all things a fair grader would ignore, and the interventions simply make ignoring them mechanical rather than hoping the model chooses to. That framing makes it easier to spot the next one — any signal correlated with your systems but not with quality is a bias waiting to be measured.

Where a judge should not be the answer

Model grading has spread into places where something cheaper and more reliable would do, and that is worth naming.

If the answer is checkable — a number, a date, a lookup with a known result, valid JSON, a tool call that either succeeded or did not — assert it. A deterministic assertion costs nothing, never drifts, and does not need calibrating. Using a model to decide whether 47 equals 47 is a surprisingly common waste.

The judge earns its place on open-ended output: was this summary faithful to the source, was this reply appropriate in tone, did this explanation actually answer the question. Those are the cases where writing assertions is impossible and human review does not scale.

And the judge is one layer. Scoring the final answer tells you nothing about how the agent got there, which is why evaluation has to cover the trajectory rather than just the output — a correct answer produced through two tool calls that should never have happened is a failing run that any judge would pass.

A related decision worth making deliberately: whether to grade pairwise or with an absolute score. Pairwise comparison — is A better than B — is what the position-bias research addresses and is more reliable, because relative judgements are easier than absolute ones for models and humans alike. Absolute scoring on a one-to-five scale is more convenient for dashboards and considerably less stable, since the meaning of a four drifts between runs and between rubric revisions. If you need a number to trend over time, derive it from pairwise results against a fixed reference set rather than asking for a score directly.

Calibrate LLM as a judge evaluation before trusting it

The step that separates teams who get value from this from teams who generate numbers is small: grade a sample by hand first.

Take fifty outputs, have a person score them, then have the judge score the same fifty and measure agreement. If the judge agrees with your human on 85% of cases you have a usable instrument. If it agrees on 60%, your rubric is ambiguous — and the fix is almost always the rubric rather than the model, because a criterion two people would interpret differently will be interpreted differently by a model too.

Keep the fifty. They become your regression set for the judge itself, and the cost of maintaining them is trivial next to the cost of discovering, six months in, that your quality metric has been measuring something slightly different ever since a provider updated a model version in March.

Re-run that calibration whenever you change the rubric, the judge model, or the model version underneath it. A judge that silently changed behaviour on a provider update invalidates every comparison you make across that boundary, and you will not notice, because the outputs are still numbers.

That is the through-line worth holding onto. A judge is a measuring instrument, and instruments need calibrating, documenting and re-checking. Treated that way it is the cheapest evaluation capability available. Treated as an oracle it produces confident scores with a systematic tilt nobody is looking for — which is why we keep the rubric, the calibration set and the judge configuration in version control alongside everything else on any AI engineering project, and why this sits squarely in quality engineering rather than in research.