The enforced protocol

This page renders bench/protocol.json — the machine-readable constraint file, written after four constraint violations in one day, each one a rule the project had already written down and then not checked before acting. Prose is read once and remembered badly; checks fire every time. The prose protocol, with the full reasoning, is at docs › benchmark-protocol.

τ²-bench pins

Headline metrics

docs/14-model-backend-benchmark.md: 'time-to-correct-result + loops-to-done, with tool-call success as a gate'. Raw reward is NOT the headline; it is binary per task and hides turn-count differences entirely.

time_to_correct_answer_s loops_to_done tool_call_success

Required run fields

Without a window, per-run energy must be reconstructed from file mtimes, and the recorder keeps 10-second history for only ~10 days.

started_at ended_at

Energy units

design doc 2.4 - joules are not a home-energy unit and do not map to /kWh tariffs.

Comparison rules

• Matched task sets. Bound-limited arms reach different depths; comparing different prefixes compares different work (clm-0037).
• Same quantisation across compared models, or the comparison measures the quant (Nemotron IQ4_XS vs peers Q4_K_M).
• Paired over identical tasks beats unpaired means at these sample sizes (clm-0038).
• State the denominator. Energy per correct answer is not energy per task is not agent-only energy (clm-0040, clm-0042).
• Compute surfaces never mix silently. NPU/CPU/GPU results follow the same benchmark format and MAY be compared for the same model across surfaces — that contrast is commentary gold (NPU prefill vs GPU, decode trade-offs) — but every displayed result carries its surface and backend (config engine + runtime.backend), purpose-fit verdicts stay within-surface, and no table row mixes surfaces without a surface column. NPU-lane candidates never inherit GPU metrics (existing schema rule).
• SMOKE VALIDITY GATE (added 2026-08-14, from the first NPU screen). A tau2 result with ZERO tool-call messages is INVALID and must never be recorded as a mean, on any surface. An agent that never acts scores the untouched-DB and COMMUNICATE points by default and can outscore a competent tool-user: npu-lfm2 scored a vacuous 1.00 this way, beating every GPU model screened, because FastFlowLM silently dropped the tools array (HTTP 200, no warning). Same do-nothing path opens whenever a tool surface fails quietly - including the HTTP-400 grammar-ceiling class. Record tool_call_messages alongside every SMOKE mean; refuse a mean without it.

Quantisations expected in a series

protocol_check_build refuses a quant matching none of these. Same quantisation across compared models is a comparison rule (the Nemotron IQ4_XS confound, clm-0047); a stray quant appearing in a series is either a typo or an unplanned comparison, and both should stop the run. UD-IQ4_XS stays listed because the IQ4_XS-vs-Q4_K_M pair is itself a measured comparison. Extend the list deliberately when a new quant enters the programme.

UD-Q4_K_M UD-Q4_K_XL UD-IQ4_XS Q4_K_M · added 2026-08-12 repeatability audit

Builds — pinned, anchored, never bumped mid-series

Minimum-required-version per model, never a fleet-wide bump mid-series. Each build is a PINNED commit with its own config records (runtime.commit); a model's minimum build lives in its candidate record. Introducing a build REQUIRES an anchor pair: the anchor workload run on both old and new binary in the same session, so the build delta is measured and nearest-comparable-config comparisons are calibrated, not assumed. Operator decision, 2026-08-10.

anchor workload: 122B UD-Q4_K_M, tau2 airline --task-ids 0 1 2 3 4 5 6 8 9 --seed 42 (subset only), plus llama-bench -p 512 -n 128,512. ~30 min. · fleet default: 3653e6d

rendered from bench/protocol.json at build time — this page cannot disagree with the check that fires