Coverage — what has actually been measured

Written 2026-08-08 in answer to a direct question: are the benchmarks done for the four models? They are not. This page exists so the answer stops depending on memory.

Headline as first written: 44 performance runs, 1 guard run, ZERO capability runs. We built a framework separating capability from performance (protocol §1) and had measured only performance.

Status 2026-08-09: the capability column is now filled — 8 arms, 4 models × thinking on/off, plus the KV-quality pair. See the update at the foot of this page. Two rows remain empty for every model (RULER, blind-judged reasoning) and one whole role has no benchmark at all (routing, clm-0034).

Per-model

Legend: ✅ done · 🟡 partial · ❌ not started · ⛔ failed

⛔ Read this before using any capability number above. The identical stock q8_0 configuration scored 0.60, 1.00, 0.60 on three consecutive runs (clm-0036). The noise width of this 5-task harness is ~0.40 — the same size as nearly every gap in the table. Three tasks always pass and two are coin-flips, so a 5-task mean is really two Bernoulli trials and can only take the values 0.60/0.80/1.00.

Marked (real) are the categorical outcomes that survive: the 122B’s failure to TERMINATE with thinking on, reproduced across four runs; and gpt-oss’s 0.00, a total failure alongside its independently reproduced staleness bug. Everything marked ±noise is a smoke test, not a ranking. The domain ships 50 tasks; these used 5.

The 122B is thoroughly characterised on performance. The other three have exactly one configuration measured each — f16 KV, flash-attn on, depths 0/4096/32768 — plus a guard. Six rows apiece. That is enough to rank them and nothing more.

What “not started” actually means for capability

This is the larger gap and it is structural, not incidental:

• τ²-bench — a smoke test passed (LiteLLM reaches the endpoint and gets a correct native tool call). No domain has been run. A real run needs the user-simulator model and the judge pinned first, because changing either silently re-baselines the whole series (protocol §8).
• RULER / long-context retrieval — never run. The guard’s needle probe is a cliff detector, not a measurement: it answers “did retrieval collapse”, not “how well does it retrieve”. Those are different questions and only one has been asked.
• Blind-judged reasoning / creativity — never run.
• KL divergence (kv-quality.sh) — never run. The corpus is now pinned (bench/CORPUS.md, wikitext-2-raw-v1 test, sha256 430983e5…) so it is unblocked, but the f16 baseline file is tens of GiB and it is genuinely an overnight job.

Consequence: clm-0022 establishes that quantised KV costs nothing in SPEED once patched. It says nothing about what it costs in QUALITY — and that is precisely the gap kv-quality.sh exists to fill. Every KV recommendation currently rests on half the evidence.

Guard status is not capability status

Three models pass the guard and one fails. That is worth exactly what the guard claims: no cliff detected. It clears the binary and hazard levers and explicitly cannot clear a lossy one. A model can pass all four checks and still be materially worse at the work — nothing here has tested that.

gpt-oss-120b is the one hard exclusion so far. It returns verbatim answers from previous requests at --parallel 1 (clm-0025), reproduced 6/6. Its throughput rows are recorded and honest; they describe how fast it generates, not whether it generates the right thing.

Honest ranking of what to do next

Rewritten 2026-08-09. Items 1 and 3 as originally written are now done — struck rather than deleted so the ordering can be judged in hindsight.

0a. Head-dim-controlled KV-quant sweep (added 2026-08-13): thefrontierlab.ai’s nine-model self-comparison found two models with byte-identical KV footprints (Qwen3-Coder-30B, Qwen3.5-122B) gaining +30.8% vs +4.5% decode from KV quant; his candidate explanations (active-param ratio, head-dim 128-vs-256 kernel behaviour) are untested. We hold models spanning both head-dims — a controlled sweep answers his open question and extends clm-0051. He publishes replications with build commits attached; ours can carry the tau2-correctness + Wh-per-answer dimensions his throughput-only series lacks. 0b. Register the BailingMoE3 PR build (added 2026-08-13): ling-30-flash’s screen recorded arch bailingmoe3 unknown on all registered builds. thefrontierlab.ai runs Ling via the open llama.cpp BailingMoE3 PR (his build at commit 017a7247). Build + register it to unblock the screen; also cross-check his report that Ling’s MTP draft path stalls at first token on this stack. 0. The gfx1151 backend-at-depth decider (added 2026-08-12): ROCm vs Nathan’s Vulkan fork (github.com/Nathanw1014/strix-halo-llamacpp, or CachyLlama’s port of the same patches) at d≥65536 on a model we hold. An r/LocalLLaMA thread (DeepSeek V4 Flash on this exact silicon) staged the fight and both sides brought data — Vulkan ahead at short depth on-chip, ROCm ~3.5× ahead at 65k depth but measured on a different GPU family — and nobody ran the decisive cell. Our clm-0046 already shows backend claims are depth-statements; this extends it from a cherry-picked PR build to the fork ecosystem people actually use, on the exact chip. Novel public result; nobody else has the rig configured for it. Secondary probes from the same thread: silent-CPU-fallback wattage signature (the fork’s v0.6 stride bug dumped 43 layers to CPU while GPU-util looked healthy — wall power would have caught it), and the quantised-drafter GGUF-header fix (halves speculative-draft footprint if real).

1. kv-quality.sh on the 122B — DONE, though not by the planned route. The KL instrument was broken (clm-0029); τ² task success answered it instead. f16 1.00 against q8_0 0.60 (clm-0035).
2. Fill the performance matrix for the other three — KV quant and flash-attn arms, and depth beyond 32k. Still the largest cheap gap; roughly one sweep each. Now more interesting than it was: q8_0 costs quality on the 122B, so the other three need their own answer rather than inheriting it.
3. One τ²-bench domain on the 122B — DONE, and extended to all four models with both thinking arms.
4. Re-run q8_0 KV on the PATCHED build — the new top item. clm-0035’s quality drop is measured on stock. If the dequant patch restores quality as well as speed, that changes the recommendation completely; if it does not, quantised KV is off the table for agentic work regardless of its speed.
5. Diagnose gpt-oss staleness — cache_prompt: false is ruled out. Now doubly worth doing: it scores 0.00 with thinking off, and the staleness bug is the obvious suspect for why.
6. A routing benchmark — the gap clm-0034 exposed. Nothing in this matrix measures the orchestrator role, and the 35B’s poor executor scores say nothing about it.
7. ROCmFP4 on the 35B (clm-0026) — worth ~27%, but a third fork to carry, and the 35B’s capability numbers make it a weaker candidate to invest in.

Update 2026-08-09 — what changed in this section

Capability is now two rows per model, not one. clm-0033 found thinking is a NET NEGATIVE on the 122B — 3/5 with it against 5/5 without, same reward where both complete. There is no reason to assume that generalises, so every model gets both arms, and the ON arm is probed for reasoning_content first so an unsupported model is recorded as such rather than silently producing a duplicate OFF run.

KV-quant quality changed instrument. kv-quality.sh is built on llama-perplexity --kl-divergence, which returns PPL 532 on English repeating every four sentences (clm-0029) — the tool is broken on this build, so no amount of care with the corpus helps. τ²-bench task success is the working substitute: coarser, but it measures quality where it actually matters.

First new number: Qwen3.6-35B-A3B scores 0.40 mean reward with thinking off, having completed all 5 tasks — against the 122B’s 1.00. It always terminates and is often wrong, which is the opposite failure mode to the 122B’s (deadlocks on hard tasks, correct when it finishes).

⚑ But be careful what that rules out (clm-0034). tau2 measures TASK EXECUTION under adversarial pressure. A reflex or orchestrator tier does triage, routing and brief-writing — a different and lower bar. A comparable project runs a 12B orchestrator successfully on exactly that split. So 0.40 rules the 35B out as an executor of hard agentic tasks; it says almost nothing about the routing role, and we have no benchmark for routing at all. That is a genuine gap in this matrix, not just an unmeasured cell.

Results — the full 8-arm matrix (2026-08-09, clm-0035)

⚑ Correction: thinking is per-model, and I generalised from one model

clm-0033 measured the 122B, found thinking a net negative, and I carried that forward as the likely general case — including flagging Warden’s thinking=high synthesis cron for reversion on that basis. That flag was premature. Thinking improves the other three models, substantially: Nemotron 0.60 → 1.00, gpt-oss 0.00 → 0.60, 35B 0.40 → 1.00.

The 122B is the outlier, not the pattern. Any global setting of this lever is wrong for three models out of four whichever way it is set — it has to be measured per model, and Warden’s cron should be judged on the model it actually runs.

Two results worth separating out

Nemotron-3-Super with thinking on is the only configuration that both scores 1.00 and completes every task. The 122B matches the score but deadlocks on 2 of 5. Nemotron is the slowest model in the set (17.51 tok/s decode) — so this is a real speed-versus-reliability trade, not a dominated option, and it is the first evidence that the 122B is not automatically the right default for agentic work.

Quantised KV costs quality, not just nothing. Same model, same tasks, only the cache type differing: f16 1.00, q8_0 0.60. The failure mode is legible — the two failed tasks ran to max_steps at 42 and 41 turns rather than answering wrongly. They failed to terminate. That closes clm-0022’s open half from the opposite direction to the one I expected: q8_0 is free in speed once patched, and expensive in reliability.

⚠ Caveat that limits all of the above: every cell is n=5 in one domain, self-play, deterministic scoring, read paths only. A 0.40 swing is two tasks. These numbers are strong enough to redirect what gets measured next; they are not strong enough to settle a production placement on their own.