The enforced protocol
This page renders bench/protocol.json — the machine-readable constraint file, written after four constraint violations in one day, each one a rule the project had already written down and then not checked before acting. Prose is read once and remembered badly; checks fire every time. The prose protocol, with the full reasoning, is at docs › benchmark-protocol.
τ²-bench pins
| pin | value | why |
|---|---|---|
| user simulator | openrouter/anthropic/claude-haiku-4.5 | Pinned independent simulator per clm-0043 and methodology-lessons 3: different family from every local candidate, dated slug, temperature 0. Served via OpenRouter (PAYG, operator account, 2026-08-11); first probe routed to Amazon Bedrock, so routing is enforced account-side: the EZAxis Provider Limit guardrail in OpenRouter allows only Anthropic (set by the operator 2026-08-11, verified by probe: provider now reports Anthropic where the first probe hit Bedrock). Supersedes the 122B self-play pin, which was only ever satisfiable when the agent WAS the 122B. Adequacy to be verified by the sim-sensitivity pair (haiku vs sonnet on the seeded anchor subset) before the cross-model series is trusted. |
| simulator temperature | 0 | determinism — the simulator is scaffolding, and a noisy simulator is a noise source in its own right (methodology lesson 3) |
| max steps | 200 | tau2 default. A lower cap sits above one model's whole turn distribution and slices through another's, biasing comparison toward terse models (clm-0039). |
| max concurrency | 1 | llama.cpp #25992 leaks responses across requests on gfx1151 HIP at --parallel > 1; and tau2 defaults to 3, which produces infrastructure_error against a --parallel 1 server. |
Headline metrics
docs/14-model-backend-benchmark.md: 'time-to-correct-result + loops-to-done, with tool-call success as a gate'. Raw reward is NOT the headline; it is binary per task and hides turn-count differences entirely.
time_to_correct_answer_s loops_to_done tool_call_success
Required run fields
Without a window, per-run energy must be reconstructed from file mtimes, and the recorder keeps 10-second history for only ~10 days.
started_at ended_at
Energy units
design doc 2.4 - joules are not a home-energy unit and do not map to /kWh tariffs.
| allowed | forbidden |
|---|---|
| Wh · kWh · mWh · pence · W | joule · joules · J/token · kJ |
Comparison rules
- Matched task sets. Bound-limited arms reach different depths; comparing different prefixes compares different work (clm-0037).
- Same quantisation across compared models, or the comparison measures the quant (Nemotron IQ4_XS vs peers Q4_K_M).
- Paired over identical tasks beats unpaired means at these sample sizes (clm-0038).
- State the denominator. Energy per correct answer is not energy per task is not agent-only energy (clm-0040, clm-0042).
- Compute surfaces never mix silently. NPU/CPU/GPU results follow the same benchmark format and MAY be compared for the same model across surfaces — that contrast is commentary gold (NPU prefill vs GPU, decode trade-offs) — but every displayed result carries its surface and backend (config engine + runtime.backend), purpose-fit verdicts stay within-surface, and no table row mixes surfaces without a surface column. NPU-lane candidates never inherit GPU metrics (existing schema rule).
Quantisations expected in a series
protocol_check_build refuses a quant matching none of these. Same quantisation across compared models is a comparison rule (the Nemotron IQ4_XS confound, clm-0047); a stray quant appearing in a series is either a typo or an unplanned comparison, and both should stop the run. UD-IQ4_XS stays listed because the IQ4_XS-vs-Q4_K_M pair is itself a measured comparison. Extend the list deliberately when a new quant enters the programme.
UD-Q4_K_M UD-Q4_K_XL UD-IQ4_XS Q4_K_M · added 2026-08-12 repeatability audit
Builds — pinned, anchored, never bumped mid-series
Minimum-required-version per model, never a fleet-wide bump mid-series. Each build is a PINNED commit with its own config records (runtime.commit); a model's minimum build lives in its candidate record. Introducing a build REQUIRES an anchor pair: the anchor workload run on both old and new binary in the same session, so the build delta is measured and nearest-comparable-config comparisons are calibrated, not assumed. Operator decision, 2026-08-10.
anchor workload: 122B UD-Q4_K_M, tau2 airline --task-ids 0 1 2 3 4 5 6 8 9 --seed 42 (subset only), plus llama-bench -p 512 -n 128,512. ~30 min. · fleet default: 3653e6d
| build | what it is |
|---|---|
| 3653e6d | stock baseline, all pre-Aug-10 runs |
| ce7689f | kvfix — stock + KV dequant patch (clm-0022) |
| min-62bf73d | muse-glimmer support (llama.cpp PR 26841); building |
| 77a9a66eb | dual-backend HIP+Vulkan single binary with PR 26856 bf16-FA cherry-pick — the clm-0046 backend-comparison build. Added 2026-08-12 (repeatability audit): it had produced published numbers without being a known build. |
| 3be50ccc | Nathanw1014/strix-halo-llamacpp v0.6.1 portable tarball (fork branch strix-halo-vulkan, Vulkan only, bundled RADV Mesa 26.3.0-devel + libdrm 2.4.134) - backend-decider arm C. Calibration: matched pp1024/tg256 depth cells vs stock 3653e6d ROCm and Vulkan run in the same session (backend-decider suite) serve as the anchor pair for this Vulkan-only build; the 122B tau2 anchor does not apply. Added 2026-08-12. |
rendered from bench/protocol.json at build time — this page cannot disagree with the check that fires