Why public leaderboards mislead, and how to build a test set that actually decides.
Public leaderboards answer a question you did not ask: how a model performs on a general academic benchmark. Your question is how it performs on your documents, in your domain, on your task.
Why leaderboards mislead
Benchmark tasks are broad and often partly present in training data. A model can top a reasoning leaderboard and handle your industry's terminology poorly, or excel on English prose and struggle with the mixed-language input your customers actually send.
Build a small, honest test set
A hundred real examples with agreed correct answers is enough to separate models decisively. Draw them from actual production traffic, include the awkward cases, and write down what a correct answer looks like before running anything. Keep this set private so no vendor can tune to it.
Measure the whole cost, not just accuracy
Latency at your prompt length, cost per thousand requests at your volume, and behaviour under load matter as much as quality. A model two points better and four times slower may be the wrong choice for a live chat and the right one for overnight batch work.
Re-run the set when a new model version ships. Providers update models, and behaviour on your task can change without any announcement that mentions it.
Want this for your business?
Let's talk about how we can help you build and grow.
