Docs › state-of-knowledge

State of knowledge

The current-state synthesis, organised by subject. For history and process, see the individual claims and docs/methodology-lessons.md. Provisional = confounded or underpowered; treat as a lead, not a conclusion.

Models

Qwen3.5-122B-A10B (MTP)

The incumbent. Decode 21.90 tok/s at empty context, 18.16 at 32k (clm-0025). Passes its capability guard and holds up on agentic work: 0.545 mean reward on tau2 airline at n=22 — not the earlier n=3/n=5 figures of 1.00, which were biased toward easy tasks (clm-0037, superseding clm-0030). On tasks it completes, it is fast: 19 turns / 2.0 min median to a correct answer, materially quicker than Nemotron on the same tasks (clm-0039, provisional — cross-model, unpinned simulator, clm-0043). Thinking is a net negative for this model specifically: OFF solves 5/5 in ~10 min, ON solves 3/5 and deadlocks indefinitely on the other two (clm-0033). The KV-dequant patch recovers up to +70.3% decode at 204,800 context, the depth production actually runs at — stock was running at 59% of achievable speed (clm-0022). Cheapest model measured per correct τ² answer: 6.81 Wh f16, 9.48 Wh q8_0 (clm-0042).

Nemotron-3-Super-120B-A12B

Slowest model measured — 17.51 tok/s decode (clm-0025) — but the most reliable completer with thinking on: 5/5 tau2 airline tasks at reward 1.00 (clm-0035, the categorical-completion part; the numeric 0.60→1.00 delta itself is retracted, clm-0036). On reward alone it is statistically indistinguishable from the 122B at scale (0.625 ±0.237, n=16, vs 0.545 ±0.208, n=22 — provisional, clm-0039, clm-0043), but needs 37% more turns and 85% more wall-clock to reach the same correct answers, and costs 87% more electricity per correct answer (12.75 Wh vs 6.81 Wh, clm-0042). Runs at IQ4_XS against the 122B’s Q4_K_M — an unresolved quant confound on every comparison involving it (clm-0039).

Qwen3.6-35B-A3B

Fastest decode of the field at empty context (51.01 tok/s, 2.3x the 122B, clm-0023), and holds speed at depth proportionally as well as the 122B does (17% loss to 32k, both models, clm-0023). Carries no MTP layers, so that lever is unavailable (clm-0026). Draft-free ngram speculation is nearly worthless on varied prompts here — 1.11x with 29% CV (clm-0026), far below a community report’s 216 tok/s solo figure (clm-0010, unreconciled — different quant finetune and prompt regime, clm-0026). The tool-call grammar ceiling once attributed to this model class does not reproduce; the real constraint is context size, not the grammar builder (clm-0024, correcting clm-0023).

gpt-oss-120b

Fastest decode of the four at empty context (55.45 tok/s) but the least bandwidth-efficient relative to its active-parameter count (34-62% realised across the field; gpt-oss itself 62%, the 35B only 34%, clm-0025). Not currently usable: returns stale answers from a previous request after the first, at --parallel 1, reproduced 6/6 and not explained by prompt caching (clm-0025). Requires flash attention to run at all. On tau2 with thinking off it scores 0.00 (total failure); thinking on brings it to 0.60 (clm-0035, categorical part only — the numeric deltas are not established, clm-0036).

Screened-but-new candidates

Acquired 2026-08-10, not yet benched — one line each:

  • Nemotron-Cascade 2 (30B/3B active) — reasoning-per-active-parameter outlier, gold-medal competitive-programming claims; text-only, vision audit closed (candidates/cascade2-30b).
  • Gemma 4 26B-A4B — claimed ~85 tok/s reflex-tier candidate; vision-capable, one mmproj download from vision-ready (candidates/gemma4-26b).
  • GLM-4.7-Flash — agentic/coding lineage, ~40 tok/s reported on ROCm; text-only (candidates/glm-47-flash).
  • LFM2-24B-A2B — the speed-vs-quality probe for the field, large prefill advantage; text-only (candidates/lfm2-24b).
  • Qwen3.6-27B-MTP — dense architecture, where dense-MTP’s larger 1.4-2x speed-up (against 1.15-1.25x for MoE) applies; regressed to acquired pending re-screen on this box (candidates/qwen36-27b-mtp).

Levers

KV quantisation + the dequant patch

Stock quantised KV costs decode speed at depth (up to 56% at 131k) and the cost grows with depth; a single cherry-picked patch removes it entirely and slightly reverses it (+70.3% at 204,800, clm-0022). The workload decides whether q8_0 is worth using, not the flag (clm-0041, clm-0042): on fixed-token throughput work it is now the cheapest option once patched (85.3 Wh vs f16’s 89.6 Wh at 200k context, clm-0041); on agentic work with variable turn counts it costs +39% more turns and +39% more energy per correct answer than f16, unpatched (clm-0038, clm-0042) — because it does more total work, not because it draws more power. Whether the patch removes the agentic penalty too is untested and is the highest-priority open run (see below).

Flash attention

Not a hazard on this hardware’s HIP backend — a claimed dense-model prefill cliff (clm-0017, community, Vulkan) does not reproduce on ROCm, where a dense 27B reaches 214.66 tok/s prefill at 32k, 5.2x the broken stock-Vulkan figure at 65k (clm-0021). gpt-oss-120b requires flash attention to run at all (clm-0025).

Speculation (ngram, MTP)

MTP delivers 1.45x on the 122B (clm-0026 citing clm-0022’s sibling work) but is throughput-negative at batch per one community source (clm-0010). Draft-free ngram speculation is close to worthless on varied, non-repetitive prompts (1.11-1.17x with 29-38% CV, clm-0026) but strong on repetitive/structured content — 1.87x on 500 IFEval prompts community-reported (clm-0016), and a further community report of 3x on code-editing content that remains unreconciled with our own measurement (clm-0010, clm-0026). Multi-slot ngram pools inflate throughput when prompts are identical across slots — a measurement hazard, not a capability gain (clm-0016).

Parallel hazard

--parallel 4 destroys long-context needle retrieval on this build — 0/8 against 8/8 at --parallel 1 — while throughput reports nothing wrong, so no performance benchmark would catch it unassisted (clm-0019). This rules out simply copying community multi-slot configurations that report higher aggregate throughput (clm-0026).

Energy

Wall-metered at the smart plug, whole-box (clm-0040, clm-0041, clm-0042). Idle floor is 10.1 W (idle-aihydra-2026-08). Under τ² inference the 122B draws ~154 W mean, Nemotron ~167 W (clm-0040). The correct denominator is per-task energy on a matched task set, not whole-arm totals — arm-level figures overstated cost by roughly 10x because most of an arm’s wall time was model loading or non-scoring tasks (clm-0042, superseding clm-0040). Per correct τ² answer: 122B f16 6.81 Wh (0.21 p), 122B q8_0 9.48 Wh (0.29 p), Nemotron 12.75 Wh (0.39 p) at 30.3 p/kWh grid import (clm-0042). Self-play overhead (the model generating both agent and user-simulator turns) inflates these figures by an estimated 20-25%; correcting for it widens the gaps rather than changing the ranking (clm-0042). On fixed-token throughput work the picture inverts: the KV-dequant patch cuts energy 42% at 200k context and makes patched q8_0 the cheapest option (clm-0041) — energy findings do not transfer between fixed-length and emergent-length workloads.

Open questions

  • Does the KV-dequant patch remove the agentic turn penalty, not just the speed penalty? The obvious next run after clm-0038/clm-0041/clm-0042 — running tonight.
  • Cross-model comparisons (122B vs Nemotron, thinking on/off) need re-running under a pinned user simulator. Currently confounded by self-play (clm-0043); see docs/methodology-lessons.md §3 for the fix.
  • A clean Vulkan-vs-HIP comparison on identical commits. A community report shows Vulkan 55% faster decode at depth, but on a 43-build gap with unsubmitted patches on one side only — not attributable to the backend as reported (clm-0031, confidence lowered to low on this basis).
  • A routing benchmark does not exist. A 12B orchestrator was observed routing competently in an unrelated project, but nothing in this project’s coverage matrix measures the routing/orchestrator role (clm-0034, docs/coverage.md).