How to evaluate an LLM: is it better, or did it get lucky?

What you're seeing

The leaderboard is five horizontal bars sorted by observed score. Every run, each model answers a fresh set of questions (200 by default) and the bars slide into their new order. The white whisker through the end of each bar is the 95% interval for that score. The dashed tick is the model's true pass rate, which you can hide with the toggle. Nothing about the models changes between runs. Only the questions do.

The grid on the left (a strip on narrower screens) is the leading model's questions for the current run, one cell each, bright for a pass and dark for a fail. It is there to make one thing concrete: a score is a count. Change which questions get asked and the count changes.

The histogram along the bottom collects one sample per run: the observed score of the truly best model (A) minus the observed score of the truly second best (B), in points. The dashed line is the running mean, the two faint bands are one and two standard deviations around it, and the hatched region left of zero holds every run where the worse model scored higher. At the defaults that region fills with roughly 30% of the runs.

Below the leaderboard is a decision readout for a team that has to pick between A and B. Under HIGHER SCORE WINS the team switches to whichever scored higher this run. Under INTERVALS MUST NOT OVERLAP the team only switches when the two whiskers are clear of each other, and otherwise makes no decision. WRONG is the share of runs where the rule picked the truly worse model. UNDECIDED is the share where it declined to pick.

A few things to try:

• Leave the defaults and press Play. Watch TOP1 in the stats. The truly best model only ranks first in about half the runs.
• Drag QUESTIONS to 5,000. The whiskers shrink to about a point each side, the histogram tightens around +2, and the hatched region empties out.
• Set SPREAD to 0. All five models are now identical. TOP1 settles near 20%, which is pure chance for five equal coins, and the leaderboard still produces a confident-looking order every run.
• Switch the rule to INTERVALS MUST NOT OVERLAP. WRONG drops to about zero and UNDECIDED climbs past 95%. The conservative rule is almost never wrong because it almost never commits.
• Press Rerun. One run plays out slowly. The whiskers start wide and narrow as questions come in, and the order usually changes hands a few times on the way.

How it actually works

Each model is a coin with a fixed pass rate: 82, 80, 79, 77, and 72% at the default spread. A run draws N questions for every model from a seeded random generator and flips that model's coin once per question. The observed score is passes divided by N. Each model gets its own draws, so this is the unpaired setup, the one a public leaderboard effectively uses when different teams report scores on their own samples of a benchmark.

The whisker is the Wilson score interval. For k passes out of n, with z = 1.96 for 95%:

p = k / n
center    = (p + z^2 / 2n) / (1 + z^2 / n)
halfwidth = z * sqrt(p(1 - p) / n + z^2 / 4n^2) / (1 + z^2 / n)
interval  = center - halfwidth  to  center + halfwidth

Wilson is a better choice than the textbook normal interval for pass rates because it never spills past 0 or 100% and it stays honest at small n and extreme scores. For 160 passes out of 200 it gives 73.9% to 85.0%, and the sim asserts that exact result on load.

The histogram bins are fixed in points for the whole series, chosen from the expected spread for the current question count, so the axis does not rescale while runs accumulate. Mean and standard deviation come from the collected samples, not from a formula. Ties for first place are broken at random with the same seeded generator, because a fixed tie order would quietly bias TOP1. Two loads of the page play out identically.

The two decision rules are deliberately simple. HIGHER SCORE WINS picks the larger count and flips a coin on a tie. INTERVALS MUST NOT OVERLAP refuses to pick unless the whiskers are clear of each other. That non-overlap test is stricter than a proper two-sample test, which is why the readout labels it conservative. It errs on the side of making no call.

What is a confidence interval on a benchmark score?

A confidence interval on a benchmark score is the range of true pass rates that could plausibly have produced the observed score, given how many questions were asked. A 95% interval means the procedure that built it captures the true rate in 95% of evals. That is a statement about the procedure's calibration across many evals, not a 95% bet on any single one. For an observed score of 80%, the whisker each side is:

• 100 questions: about 7.8 points.
• 200 questions: about 5.5 points.
• 1,000 questions: about 2.5 points.
• 5,000 questions: about 1.1 points.

The pattern is a square root. Four times the questions buys half the interval, and a hundred times the questions buys a tenth. That is why going from 200 questions to 400 helps less than people expect, and why the last point of precision is expensive. A leaderboard gap smaller than the whisker is not a result. It is a coin flip wearing a decimal point.

How many test cases do I need for an LLM eval?

Enough to resolve the gap you care about. For two models scored on independent question sets, both near an 80% pass rate, the standard deviation of their score difference is sqrt(2 x 0.8 x 0.2 / N). If you want the real gap to be at least two of those standard deviations, so that the worse model wins by luck only a few percent of the time, the question count per model works out to roughly 1.28 divided by the gap squared:

• A 10-point gap: about 130 questions.
• A 5-point gap: about 500 questions.
• A 2-point gap: about 3,200 questions.
• A 1-point gap: about 12,800 questions.

Those are the numbers behind the sim's default. At 200 questions and a 2-point true gap, the worse model scores higher in about 30% of runs. At 5,000 it is under 1%. The next step a careful team takes is the paired comparison: grade both models on the same questions and count only the questions where they disagree. Because the shared questions cancel out, a paired test resolves the same gap with far fewer questions. The sim does not build it. It stays on the unpaired case because that is the one leaderboards show you.

Why do AI model rankings change?

Partly because models get updated, prompts get rewritten, and benchmarks get revised. But the sim removes all of that and the rankings still change, because five models within a few points of each other on a few hundred questions are inside each other's intervals. At the defaults the truly best model ranks first only about half the time, and the models in second through fourth trade places constantly. With the spread at zero, where all five are identical, the sim still prints a crisp first-to-fifth order every run, and the truly best model lands on top one run in five. The ranking is real. The information in it is not.

Are AI benchmarks reliable?

They are reliable for what they measure, on the questions they ask, inside their interval. The trouble starts when a score is read as a property of the model instead of a measurement of it. And the finite question set is only one noise source. The sim says so on screen, and it matters: real evals also move with the prompt wording, with the sampling temperature used to generate answers, with grader error when an LLM acts as judge, and with test questions that leaked into training data. None of those show up in the whisker. The interval is a floor on your uncertainty, not a ceiling.

A careful eval report states the question count, shows the interval, re-runs with different seeds and prompt variants, compares models on the same questions, and keeps a private test set the training pipeline has never seen. If the grader is a model, probe it for bias first. Our LLM as judge walkthrough covers how one answer gets graded and where the judge's blind spots are. If you run the generator at a nonzero temperature, expect the same model to produce different answers on the same question, which is one more thing to average over. Our temperature explainer shows exactly what that knob does to the output distribution.

The habit that fixes most of this is cheap: before you believe a gap, ask how many questions stand behind it. A single benchmark number deserves the same trust-but-verify treatment as any other confident output from a model.

Key takeaways

• A benchmark score is a sample. Run the same eval on a different draw of questions and the score moves, with nothing about the model changed.
• The 95% interval on an 80% score is about 8 points each side at 100 questions, 2.5 at 1,000, and 1.1 at 5,000. Four times the questions buys half the interval.
• At 200 questions, a model that is truly 2 points worse scores higher in about 30% of runs. At 5,000 questions that drops under 1%.
• A leaderboard gap smaller than the interval is not a result. Refusing to decide when the intervals overlap costs you decisions, not accuracy.
• The finite question set is only one noise source. Prompt wording, sampling temperature, grader error, and contaminated test data all add more, and none of them show up in the whisker.

Frequently asked questions