Method
Why any of this should be believed. Every published number is wall-measured on the lab's own hardware, carries its provenance and confidence, joins to the run and configuration that produced it, and is corrected in public when it falls. The pages below are the machinery: the protocol that constrains how measurements are made, the standing rules extracted from measurement failures, and the corrections the process has already produced.
The protocol
Two measurements, not one: capability is what a model can do — slow, expensive, run rarely; performance is how fast it delivers that — cheap, run often, and only meaningful behind a guard, the cheap cliff-detector run at every performance configuration. What makes that split legal is a classification of every configuration lever by how it can break capability, and a machine-readable constraint file the harness checks before every run — prose is read once and remembered badly; checks fire every time.
- The enforced protocol — bench/protocol.json rendered: pins, units, comparison rules, known builds
- The full protocol document — the prose version, with the reasoning behind every rule
fleet default build 3653e6d ·5 known builds, each pinned with an anchor pair · user simulator pinned to openrouter/anthropic/claude-haiku-4.5 ·5 comparison rules enforced
Corrections — the record correcting itself
Corrections live in the Log timeline with every other entry, badged inline — visible, not enshrined. The registry below them is a generated filter of that same stream: every retracted or superseded claim, what was wrong, and what fixed it.
- The corrections registry — 3 withdrawn claims, kept citable with their replacements
SUPERSEDED by clm-0042's per-task measurement, which found the true energy cost roughly 10x lower — this run's figures are whole-arm totals padded by model loading and non-scoring tasks, not the model's actual energy per answer, and must not be used to rank models. As measured here: a correct τ² answer cost 78.0 Wh on the 122B with f16 KV, 100.1 Wh on Nemotron, and 115.3 Wh on the 122B with q8_0 KV, but only 8-27% of each arm's wall time fell inside a scored task. aihydra's power envelope stands on its own: idle 10.1 W, 150-168 W under inference, peaking at 218 W.
SUPERSEDED: the Pass^1 = 1.000 reported by this run came from a 3-task subsample biased toward the domain's easiest tasks — the other two of the original five never terminated and were excluded as infrastructure errors. The sustained score across a realistic sample is clm-0037's 0.545 (n=22), which is the number to cite for this model on tau2 airline. This run was nonetheless the project's first genuine capability measurement rather than a throughput number, and it proved the harness and scoring path work.
RETRACTED: the per-model reward rankings and quantised-KV cost reported by this run do not hold — they came from 5-task tau2 arms whose ~0.40 run-to-run noise and 40-step cap bias were only characterised afterward (clm-0036), so the reward numbers below are not usable. The corrected 122B score is clm-0037's; the corrected cross-model comparison is clm-0039's. What survives is categorical, not scored: the 122B fails to terminate some tasks with thinking on, and gpt-oss fails the domain outright.
Lessons — standing rules with their scars
Rules extracted from measurement failures, not a diary of them. Each cites the claim it was drawn from, so the worked diagnosis is one click away.
- 1. A small-n benchmark can silently become a coin flip
- 2. Sanity-check an instrument against known bounds before trusting its output
- 3. Pin the user simulator, or cross-model comparisons are confounded
- 4. A bound chosen for convenience is not a property of the system
- 5. Match the energy denominator to the question
Essays — the long-form record
Instruments
The bench scripts are the instrument set — each enforces part of the protocol rather than trusting a reader to remember it. Energy is read from a wall meter's cumulative kWh counter, differenced across each run's window; never modelled, never joules.
scripts: cache-trace-probe.py cache-trace.sh capability-probe.sh env-lock.sh export-tool-schemas.sh grammar-ceiling.sh guard-probe.py guard.sh kv-quality.sh preflight.sh protocol-check.sh runmeta.sh spec-ab.sh sweep.sh tool-calling.sh
meters: TP-Link smart plug via Home Assistant AI Hydra energy/power sensors via Home Assistant (counter-difference on sensor.hardware_ai_hydra_energy; peak/mean drawn from sensor.hardware_ai_hydra_power) · corpus pinned in bench/CORPUS.md
Contributions — upstream work, fork-carrying disclosed
- con-0001 · ggml-org/llama.cpp fork — carrying Disk slot save/restore silently loses all prompt reuse on hybrid/recurrent models because context checkpoints are never persisted. Fixed with a .ckpt sidecar and pushed to headbouyJB/llama.cpp@fix-25913. Independently confirmed working by two community testers. A third-party PR (#26004) fixes the same bug by appending checkpoints inside the save file instead; production carries our fork until one of the two lands upstream. · upstream
carrying marks a fork in production use but not merged upstream — a reproducibility hazard disclosed on every configuration that depends on it, not a badge of honour