Benchmark
Compare models using Aperant tests and inspectable external benchmarks.
Measured 24 Sept 2026 — newest run 4 weeks old. Fetched 24 Oct 2026, 10:59 UTC, refreshed at most every 5 min.
Last run · Each score is a single judged run · every number below is the Aperant app's, published with the run
Overall = mean of the topic scores — every topic weighted equally (12 topics), frozen before the run started. A topic is weighted the same whether it holds one test or seven, so a large topic does not outvote a small one. A topic a model was not measured on is left out of its average, not scored zero.
Model leaderboard
| # | Model | Score | Passed | Time/task | Cost/task |
|---|---|---|---|---|---|
| 🏆 | claude-haiku-4-5-20251001automaticanthropic-claude-subscription-cli | 67 | 2/3 | 2.9m | sub |
Every number above is the Aperant app's, published with the run; this page recomputes nothing. Pass or fail is decided by a script wherever one exists and by the judge otherwise — each test says which. Click a model to see every test it ran, and a test to see what it answered.
Raw run data
Local model runs
Self-reported| Subject | Test | Pass | Wall time | Cost | Input | Output | Cache read | Cache write | Total | Verified |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5claude-haiku-4-5-20251001 | 02_coding/01_calculator | pass | 42.5s | $0.03 | 3,521 | 5,329 | 0 | 0 | 8,850 | Self-reported |
| Claude Haiku 4.5claude-haiku-4-5-20251001 | 02_coding/06_task_scheduler | pass | 1.4m | $0.06 | 3,599 | 10,484 | 0 | 0 | 14,083 | Self-reported |
| Claude Haiku 4.5claude-haiku-4-5-20251001 | 02_coding/07_code_oss_tab_clusters | fail | 6.6m | $0.73 | 681 | 38,001 | 3,803,529 | 81,518 | 3,923,729 | Self-reported |
How this is measured
- Where do the numbers come from?
- From the Aperant app. It computes the board when a run is published and this page shows that computation verbatim — rank, score, spread, pass count, time and cost. This page recalculates nothing, so the number here is the number in the app, by construction rather than by two copies of one formula staying in step.
- Who decides pass or fail?
- A script, wherever one exists. 2 of 3 tests are decided by a deterministic check — a regex, an exact match, a test suite — which is what 67% hard means on the tabs. On those tests the judge's number is shown as a note in the drill-in and never ranked on. The rest are the judge's opinion, and every drill-in row says which kind it is. A human override, where one exists, outranks both.
- Who is the judge?
- claude-opus-5, frozen into the run before it started. It is not a model on this board, so it is never scoring its own output.
- How many times was each test run?
- Each score is a single judged run. The spread between judged runs is published as ± beside each score, and left blank below two judged runs rather than shown as ±0.
- How is the Overall score computed?
- By the app, at publish time: every test is scored 0-100, those scores are averaged within each topic, and Overall is the average of the topic scores — every topic weighted equally (12 topics), frozen before the run started. It is deliberately not a flat average over tests: that would make a topic's influence its item count, so an ability with seven tests would outweigh one with a single test by seven to one. A topic a model was not measured on is left out of its average rather than scored zero — an unmeasured ability is not a failed one.
- Can I check any of this?
- Yes — that is the point. Open any model, then any test: you get the exact prompt, the model's full answer, the judge's reasoning, who decided the verdict, and the timing and cost. Raw run data for every published run is at the bottom of this page.
- Why five token columns instead of one total?
- Because a single total hides where the cost went. Cache reads and cache writes price differently from fresh input, so one combined number can make an expensive run look cheap.
- What this is not
- It is not a gold-standard benchmark, and it is not a claim that one model is better than another in general. It is a small, public, checkable set of tests, run on a stated date against stated builds.