Benchmark protocol

How this test bed measures things, and what it refuses to measure. Written 2026-08-04, before the boxes are back — deliberately, because a protocol designed after the first results is contaminated by them.

The governing metric is unchanged from Phase 3C: time to a correct result, with tool-call success as a gate rather than a tiebreaker. Raw tokens/second is a means. A model that decodes faster but reasons in circles is slower where it counts.

0. Preconditions — pin these or the results mean nothing

Not axes. Get one wrong and the run measures the mistake instead of the subject.

1. Two measurements, not one — capability and performance

Everything below rests on this split, so it comes before the axes.

Capability is what a model can do: retrieval at depth, tool calling, blind-judged reasoning. Performance is how fast it delivers that: prefill, decode, TTFT, and the cache and parallelism machinery around them. A full benchmark establishes capability, then finds the configuration that delivers the best performance without breaking it.

The two differ enormously in cost. A capability suite is hours; a llama-bench row is seconds. That asymmetry is the whole reason to separate them — it lets capability be measured rarely and performance often. What makes that legal is knowing precisely which levers can break capability, and how.

1a. Levers, classified by how they touch capability

Neutral is a proof, not an optimism. Correctly implemented speculative decoding reproduces the base model’s distribution exactly — the verification step guarantees it. So an unchanged quality score under MTP or ngram is the expected result, and mainly evidences that the implementation is not lossy. This is why clm-0016’s IFEval-strict 78.6% under ngram-mod is reassuring about the implementation rather than surprising about the technique; the more interesting half of that result is that it also held under ROCmFP4, which is a lossy lever.

Hazard is the class that breaks the clean separation, and it is not hypothetical: llama.cpp #25992 leaks responses across concurrent requests on gfx1151 HIP at --parallel > 1. That is a throughput setting silently corrupting output. Note that the community’s headline 121 tok/s sustained figure was measured at parallel 4 on HIP on exactly that silicon, and an IFEval score cannot clear the concern — contamination would surface as scattered per-prompt failures, indistinguishable from ordinary model error at that granularity.

Capability is a function of depth, not a scalar. A model fluent at 8k can fail at 200k. Every capability result carries depth_tokens; one without it is not a result.

1b. Three tiers

1. Capability suite — expensive, thorough, run per (model, quant, template).
2. Guard (bench/guard.sh) — cheap cliff-detector, run at every performance config. Coherence, a tool call with non-empty arguments, a shallow needle at depth, and cross-request isolation when parallel > 1. Minutes, not hours.
3. Performance sweep — run freely once the guard passes.

The guard is what makes “best performance without breaking capability” tractable. It does not certify capability; it detects cliffs. A lossy lever cannot be cleared by a guard, and the schema enforces that: inheriting capability across a quant or backend change is a build error, not a judgement call.

1c. Publish the frontier, not the winner

“Best performance without breaking capability” implies a threshold, and thresholds are judgement calls that age badly. Every config is published carrying both its capability basis and its throughput, with risk flagged rather than the row suppressed — the same pattern already settled on for OOM. A schema that refuses to record a fast-but-degraded config is the same schema that refuses to record reality.

2. Axes — and why full factorial is the wrong instinct

Five axes interact: backend × context × quant × KV-quant × MTP. Full factorial is hundreds of runs, most of them uninformative.

Sample deliberately instead:

• Backend × context is the primary grid. Not a fixed choice — ROCm was picked in June because Vulkan vk::DeviceLost-crashed at 128K, but the community quantized-KV work is Vulkan-side for prefill and ROCm-side for decode. The right backend may now differ by phase and by context, which is exactly the kind of thing an inherited assumption hides.
• Quant × MTP must be measured together, as a ratio. Published Strix Halo numbers show the MTP multiplier moving opposite to intuition — 2.44x at Q8_0 against 1.81x at Q4_K_M — because decode is bandwidth-saturated, so fewer memory passes pay more the heavier the weights. Measuring either alone gives the wrong answer about both. The transferable number is the ratio, not the absolute.
• KV-quant is now a first-class axis, not a memory-saving footnote. Community verification reports stock llama.cpp repeatedly dequantizing during inference on this silicon, with a fix worth +75% to +203% ROCm decode and q8_0 running faster than f16. If true here, “quantize the cache to save memory” becomes “quantize the cache to go faster”, which inverts the trade. See clm-0006.
• Cold vs warm is a dimension, not noise. Every row carries served: warm|cold. A suite that measures only cold prefill misrepresents production; one that measures only warm hides the cold-start cost that dominates a first turn.

3. Tiers — cheap and broad first, expensive and narrow last

Tier 0 — capability probe. Does it load, what is the max fit context, what is peak memory. Minutes. Kills candidates before they consume eval time — and produces observed_peak_gb, the field clm-0001 says we are missing. Capture peak during Tier 0 rather than as a separate exercise; it is nearly free here and expensive later.

Tier 1 — throughput sweep. TTFT, prefill and decode tok/s across 4K → 128K → 262K, per backend. Throughput on this hardware is deterministic (N=3 stddev ≈ 0), so N=3 at ≤32K as published variance evidence, N=1 above where the number does not move.

Tier 2 — cache capability trace. The operator’s addition, and the part nobody else publishes (§3).

Tier 3 — agentic quality. Expensive, so run it only on the configuration Tier 1-2 selected, at the context band we actually operate in (>100K). N≥3 minimum, because this is the only place run-to-run variance is real. Blind-isolated judging: a separate, fresh judge per response seeing only the task, the rubric and one anonymised answer.

The R1 rule. A prescriptive scorer once ranked the eventual winner 0/15 while it was producing the highest-value output in the field — the gate was measuring entity-name luck, not reasoning. No pass/fail gate is trusted until transcripts have been read. Scores propose; transcripts decide.

4. Cache capability trace — the original contribution

Everyone publishes tokens/second. Nobody publishes what their cache actually does, and on this box cache behaviour is the difference between an 8-minute turn and a 10-second one. A complete trace per config:

This is also where the cache-wall hypothesis (clm-0007) gets tested: a measured GPU-cache knee at 32-40 MB with a ~4x read-speed drop past it, offered as a candidate explanation for our 354 → 144 tok/s prefill curve. Distinct mechanism from pool utilisation — do not conflate the two.

5. What every run must carry

Non-negotiable, or the result is not publishable: config id (never a naked number), build_commit, backend, runtime.tree, chat-template version, served: warm|cold, N and the variance where N>1, and the suite version. Plus incidents: [] — a run during a thermal event or a competing process is contaminated, and that must be visible at the point of use rather than buried in a postmortem.

Suite versioning: patch = comparable, minor = comparable with a note, major = breaks the series — stated explicitly in comparable_with, never inferred from the number. The grader is a dependency: changing the judge model silently re-baselines all history, so it is a major bump even when the test content is identical.

6. Cross-host runs — the comparison nobody has

With two boxes, run the same suite and config on both and set cross_host_pair. That isolates hardware as the only variable. It is the comparison people actually want when deciding whether a second box is worth buying, and it is almost never available because most people replace rather than overlap. Run it during the overlap window or lose it.

7. Build on existing work — what to adopt rather than invent

Checked before writing anything bespoke. Most of Tiers 0-1 already exist and are better standardised than anything we would write.

The interoperability prize. Emitting llama-bench-compatible JSON makes our numbers directly comparable to everyone else’s on this silicon, and directly contributable — the standing community request is literally “reply with the numbers, with build_commit and the fa mode from the bench JSON attached”. A bespoke format would forfeit that for no gain.

What stays ours, deliberately: the cache capability trace (§3) — nobody publishes it, and on this box it is the difference between an 8-minute turn and a 10-second one; energy per correct answer, wall-metered with a stated baseline; lived production telemetry alongside controlled benchmarks, kept strictly separate; and the decision log — what displaced what, and what was rejected.

8. Task sets — adopt the standard ones, keep the bespoke ones for a reason

Distinct from §6, which is about plumbing. This is about content: which task sets to run. The rule is the same one the site is built on — the scores in the public commons are unreliable; the tasks are not. Adopting a task set while generating our own numbers against our own config fingerprint is exactly the point. It is the missing fingerprint we criticise, not the questions.

⚑ Neither benchmark tests our actual failure mode, for opposite reasons. τ²-bench uses native tool-calling but its domains are far too small (13-17 tools) to approach the ceiling; MCP-Bench has large authentic schemas but bypasses native tool-calling entirely. So the grammar-ceiling probe stays bespoke — and the cheap win is to lift MCP-Bench’s mcp_servers/ corpus as raw fixture data (28 real third-party MCP schemas) and feed combinations of them into a native tools=[...] request against llama-server, scaling until HTTP 400. That extends the manual probe from 2026-07-05 (55 tools = 200, 60 = 400) with authentic schema diversity, for about an hour’s work and none of their machinery.

⚠⚠ VERIFY BEFORE SPENDING A NIGHT ON THIS — the hybrid confound. Our 122B is a hybrid/recurrent architecture; that is precisely why the checkpoint bug (con-0001) existed at all. On such models -ctk/-ctv only affect the minority of layers that use full attention — the linear-attention layers hold no KV cache to quantise. llama.cpp issue #21385 reports q4_0 KV token-identical to f16 (BLEU 1.000) on a Qwen3.5 hybrid, attributed to only 8 of 32 layers using full attention. That both predicts a tiny delta and dilutes any measurement of it. Count how many layers actually hold a quantised KV cache before designing the matrix — if it is a small minority, the honest finding is “q4_0 is near-free on this architecture, and that is an architecture result, not a quantisation result.”

Two further traps: on ROCm only matching -ctk/-ctv types hit the fused flash-attention kernel — mismatched types fall back silently to a slower path, so keep K and V identical or you are measuring something else. And the f16 arm may not fit at 128-200K beside a 122B; if f16 only fits at 64K, the three arms are not comparable at the top length and the matrix must be redesigned around what actually loads.

Operational note: run all of this against the raw llama-server, never through llama-swap or the slotpin proxy. A multi-hour sweep will evict the production warm-lane cache and wedge the single slot.

The compounding benefit: RULER at three KV quantisation levels answers two questions at once — our own “is q8_0 safe at 200K”, and the community’s open “what does cache compression cost in quality”. Same runs, two audiences, and the second one is a standing request from people who published their half of it.

8a. Live Home Assistant: schemas yes, actuation no

No benchmark touches a live Home Assistant. Not as a safety rule but a measurement one: HA state is mutable and shared, so a suite that toggles real helpers changes the world it is measuring and cannot be re-run from the same starting state. It would also silently import HA version, MCP server version, integration availability and entity naming into every result as uncontrolled variables.

We have already paid for this lesson. The R1 task in Phase 3C appeared to measure energy-dispatch reasoning. Reading 60 transcripts showed it measured whether a model happened to toggle an entity literally named switch.dishwasher and whether the mock recognised the entity IDs it guessed — zero genuine reasoning failures across the set, and it nearly selected the wrong model. Environment coupling is how a benchmark starts scoring the environment.

But schemas are not actuation, and one probe genuinely needs the real ones. The grammar ceiling is a schema-size limit; synthetic stubs are shorter and more shallowly nested than real MCP tools and would flatter the result. So:

Capture the schemas, freeze them, mock the execution.

Export the tool definitions from the HA and Apple MCP servers once, commit them as static JSON fixtures, and replay them into a real tools=[...] array (bench/grammar-ceiling.sh --schemas <dir>). Authentic size and shape, no live dependency, repeatable — and versioned, so a later HA upgrade appears as a deliberate fixture change rather than as silent drift in a result nobody re-derived.

If HA-shaped tasks are wanted, mock the environment and treat the mock as a suite dependency: versioned, frozen, and bumped like the grader. R1’s real lesson was not “do not mock” but that the mock’s fidelity became the thing being scored.

Where live HA does belong: telemetry, not benchmarks. §2.7 already separates them — benchmarks are controlled and reproducible, production telemetry is observed and workload-dependent, and mixing them destroys both. Real Warden-touching-real-HA behaviour is a counters-and-distributions question.

9. When a run fails — which levers are safe, and which invalidate the result

Benchmarks fail mid-suite. The dangerous moment is not the failure, it is the twenty seconds afterwards when you change one flag to make it complete. Some changes cost nothing but a footnote; others silently produce a number that cannot be compared to anything, including our own earlier runs.

The rule

If the change would alter the config fingerprint, it is a NEW CONFIG, not a retry.

That is not a metaphor. The fingerprint is computed over exactly these fields — build_commit, model_filename, type_k, type_v, n_gpu_layers, flash_attn, n_batch, n_ubatch (see scripts/ingest-llama-bench.mjs). The fingerprint fields ARE the invalidation boundary. Touch one to rescue a run and you have not rescued it; you have started a different experiment that happens to share a suite name.

Safe levers — declare them, then carry on

These change how much we measure, not what:

Invalidating levers — a new config, or nothing

Reach for any of these and the old and new numbers must never appear in the same series:

Fingerprint fields: quant · KV quant (-ctk/-ctv) · -ngl / offload split · flash attention · batch and ubatch · build commit · backend · model file or revision.

Beyond the fingerprint, equally fatal:

• The grader model. Changing it silently re-baselines every judged result in history. A major suite bump even if the task content is byte-identical.
• The task set. Any edit is a suite version bump; whether it stays comparable is stated in comparable_with, never inferred from the number.
• preserve_thinking. A precondition, not a setting. Flip it and agentic results measure a template defect.
• System prompt or bootstrap. Changes the prefix, which invalidates every cache-behaviour metric outright — reuse is measured against that prefix.
• MTP on/off. A different decode path.

The grey zone — permitted, but the caveat travels with the number

Sometimes the only way to measure a thing at all is to compromise. That is allowed. What is not allowed is letting the compromise sit in a footnote while the number travels alone.

When to stop rather than push through

Three failures where continuing produces confident nonsense:

1. Empty-argument tool calls. Stop. That is the preserve_thinking signature, and every agentic number collected past it measures the template, not the model.
2. HTTP 400 on grammar build. Our tool surface exceeded the schema-size ceiling and the fallback is silent — the run appears to succeed against a different model entirely. Fix the tool count, do not record the run.
3. OOM anywhere in the sweep. On unified memory this takes the whole box down and needs physical access. Stop the matrix, record the ratio at failure — that is the calibration data clm-0001 says we lack — and only then decide what to change.

10. What we deliberately do not measure

• Completeness. Decision-relevance is the bar: what is live, what displaced what, what was rejected. A run that cannot change a decision does not need to exist.
• Correctness in production. It is not observable there. Production yields throughput, cache behaviour and cost — not quality. Say so on the page rather than implying otherwise.
• Output quality under KV quantisation — currently. This is a known gap in the community work too, and the one place we already hold an answer worth contributing: NIAH at q8_0 KV, perfect retention to ~205K including the multi-needle tier.

Compute surfaces and the NPU lane (added 2026-08-13)

The NPU lane follows the SAME benchmark format as GPU/CPU work — same tiers, same record types, same energy discipline. Screening tier for NPU candidates: FIT = loads-and-serves under FastFlowLM on the NPU (device evidence required: NPU lock, power-state transition); GUARD = the standard probes over FLM’s OpenAI-compatible endpoint; SMOKE = the standard 5-task tau2 set, pinned simulator, cliff-detector semantics unchanged.

Marking rule (operator, 2026-08-13): every result is labelled with its compute surface and backend (config engine + runtime.backend). Cross-surface comparison of the same model is welcome IN COMMENTARY — “NPU prefill is x% faster, decode slower” is exactly the interesting contrast — but purpose-fit verdicts are per-surface, and no table mixes surfaces without an explicit surface column. NPU-lane candidates never inherit GPU metrics (schema-enforced).