clm-0016

Single community source (the ACE-SABER follow-up post), self-reported, on someone else's Strix Halo. Confidence is medium DESPITE the detail, because it is one operator on one box and none of it is independently reproduced. The internal arithmetic does check out: 1.32M tokens / 3.0 hours = 122 tok/s, consistent with the stated 121. CONFIGURATION (all five are fingerprint fields, so this is ONE config point, not a decomposition): ROCmFP4 quant, HIP backend, ngram-mod speculation with shared hash pool, --parallel 4, f16 KV. THE NUMBER THAT MATTERS: **121 tok/s sustained vs a 64.8 tok/s no-speculation floor = 1.87x**. Both figures come from the same operator under the same conditions, which makes this a genuine isolation of the speculation effect rather than a headline. The author explicitly labels 121 as "the production number". WHAT THE AUTHOR DISCARDS, AND WHY IT MATTERS MORE THAN WHAT THEY KEEP: 430 tok/s (same-prompt repeat) is flagged as an artifact — only one session doing real work while three ride the ngram pool. 302-305 tok/s (4 streams, identical content) is flagged as "inflated by max pool sharing". Only 380 tok/s (single stream, real agentic session) and 121 tok/s (sustained, varied) are claimed as trustworthy. This is exactly the discipline our protocol demands and it is why this post is usable at all. ⚠ IT ALSO CONFIRMS A HAZARD WE ALREADY FLAGGED. model-phases §3a notes that ngram-mod's shared pool only works within one llama-server, so isolation and speed conflict. This data shows the pool ALSO corrupts measurement: identical content across slots inflates throughput because the slots feed each other's n-gram cache. **Any multi-slot benchmark of ngram-mod must use VARIED prompts or it measures the pool, not the model.** Our sweep.sh does not currently guard against this. PREFILL DECAY AT DEPTH — THE FINDING WITH THE MOST CONSEQUENCE FOR US: | prompt | prefill | |---|---| | 8.1k cold | 1,211 tok/s | | 2.2k blob at ~3k depth | 1,186 tok/s | | 2.2k blob at ~13k depth | 899 tok/s | | 2.2k blob at ~243k depth | **136 tok/s** | Read carefully: that last row is the MARGINAL cost of appending 2.2k tokens when 243k are already resident — roughly **16 seconds of prefill before a single token is generated**, every turn, at depth. It is not the cost of prefilling 243k from scratch. This is the strongest external validation yet of the warm-lane and disk-restore work: our measured restore was 105 ms for 44k tokens, so restore-versus-reprefill is the difference between milliseconds and tens of seconds per turn. TTFT corroborates it — 6.7 s for a cold 8.1k prompt versus **71 ms** for turn 2 on a cached prefix. QUALITY — AND WHAT THE IFEval NUMBER ACTUALLY TESTS. IFEval strict 78.6% / loose 79.8%, reported as no measurable drop. Worth being precise about what that establishes: correctly implemented speculative decoding is mathematically exact — the verification step reproduces the base model's distribution — so an unchanged score is the EXPECTED result and mainly evidences that the ngram implementation is not lossy. The more interesting half is that it also passes under **ROCmFP4**, which is a genuine quantisation change and could have cost accuracy. Note too that IFEval measures instruction-following only: it says nothing about long-context retrieval at 243k, nor about tool calling, which are the two axes we care most about. The author says more benchmarks are coming. SCOPE LIMIT — 27B GETS NO SIMILAR BOOST. Reported plainly, and the mechanism is plausible: speculation pays in proportion to how memory-bound decode is, because batch verification of drafted tokens is nearly free when you are bandwidth-starved rather than compute-starved. A3B activates 3B; a dense 27B activates all 27B and is far more compute-bound, so there is less idle bandwidth for speculation to exploit. **Prediction worth testing rather than assuming:** the benefit should scale inversely with active parameters — largest at Ling-3.0-flash (5.1B active, clm-0015), smaller but real on our Qwen3.5-122B-A10B (10B active), negligible on dense models. VULKAN, SECOND INDEPENDENT REPORT: "Vulkan? Nope, has a bug that's kicking in, HIP is the fix." This is now the second unrelated account of Vulkan misbehaving on gfx1151 with a Qwen3.6 model, alongside our own note about a backend being faster while emitting garbage. Two sources is not proof, but it is enough to make HIP the default for our sweeps and to treat any Vulkan throughput advantage on this silicon as suspect until output is verified. WHAT WE SHOULD TAKE FROM IT: a quality anchor (IFEval strict 78.6%) and a throughput target (121 tok/s sustained, 64.8 floor) that our own runs can be compared against, on the same silicon we own — provided we state the five-way config difference rather than presenting it as a like-for-like row.