Benchmarks and model evaluation: what the numbers really say

Every new model release now follows a liturgy: the benchmark table with the winning scores in bold, the charts overtaking the competitor, and in the following days the corporate committees asking whether it is time to migrate. The liturgy works because the numbers look objective, and in part they are; the problem is what they measure, and the distance between that what and the work the model will have to do in your company. A distance no public leaderboard bridges, and that can be bridged in only one way, by measuring at home.

What the leaderboards actually measure

Academic and industry benchmarks evaluate general capabilities on standardised tests: knowledge and reasoning questions, mathematical problems, programming exercises, comprehension tasks. They are precious instruments for model developers and for tracking the sector's trajectory, and they suffer from three structural limits when they become purchasing criteria.

The first is contamination: public tests end up, directly or through paraphrase, in training data, and a high score may measure memory more than capability. The second is saturation, because on the historical benchmarks the leading models now crowd within a few percentage points, a zone where differences say little and noise says a lot. The third, the most important for buyers, is domain gap: no standard test contains your documents, your industry jargon, your formats, your thresholds of acceptable error, and the correlation between general excellence and performance on your specific task exists, but it is too loose to sign a contract on.

To these adds the limit numbers cannot capture by construction: the leaderboard says nothing about latency at your volumes, cost at your traffic, behavioural stability between versions, quality in your working language, all variables that in production weigh as much as capability.

The evaluation that counts: the golden set

The alternative has a surprisingly low initial cost and a name borrowed from engineering: the golden set, a curated collection of real cases of your task, with the expected output defined by the people who actually do the work. For a typical use case, fifty to two hundred well-chosen examples suffice, covering the routine, the edge cases that hurt you and the errors you cannot afford, each with its reference answer and the criteria for judging acceptable variants.

On that set every candidate is evaluated under identical conditions, with metrics decided beforehand and tuned to the task: where the output is verifiable you automate, where it is open text you combine sampled human review with assisted evaluation, taking care to check periodically that the automatic judge judges the way you would. The result is a table no public leaderboard can give you, the competing models ranked by quality on your work, with cost and latency measured on your traffic next to them: the three columns a decision gets signed on.

The golden set, once built, keeps yielding: it is the bench on which every new model the release liturgy puts in front of you gets tested (migration becomes an afternoon of tests instead of a debate), it is the safety net for verifying that a provider update has not changed behaviours you built upon, and it is the embryo of quality monitoring in production, where the new and difficult cases met in the wild flow back to enrich it.

The discipline that follows

The operational rule we hand to clients fits in one line: no model choice or migration is decided on a public leaderboard, all are decided on the golden set. The leaderboard stays useful for one purpose only, building the list of candidates to bring to the test bench, and for that it is more than enough.

There is one practical obstacle, the initial effort of building the set, which requires time from the people who know the task, by definition the busiest ones. It is exactly the kind of work our chain covers with prototyping: in a few weeks golden set and evaluation bench get built together with the first prototype, so that the model choice is born measured and stays measurable: calendly.com/fabiolalli/zerofive, or hello@zerofive.ai. In the meantime, if a migration is being discussed in your company on the wave of the latest release, one question brings the conversation back to earth: on how many of our real cases have we tried it, and who judged the answers?