clm-0020

Qwen3.5-122B-A10B UD-Q4_K_M, aihydra, ROCm 7.1.0, llama.cpp 3653e6d (STOCK — no fork, no patches), `--load-mode none`, 3 reps. Full rows ingested as run-0008..run-0023. | depth | fa | KV | pp512 | tg128 | |---|---|---|---|---| | 0 | 0 | f16 | 312.04 | 21.68 | | 0 | 1 | f16 | 321.72 | 21.91 | | 4096 | 0 | f16 | 283.47 | 19.26 | | 4096 | 1 | f16 | 300.67 | 21.39 | | 32768 | 0 | f16 | 170.81 | **9.87** | | 32768 | 1 | f16 | 208.77 | **18.17** | | 32768 | 1 | q8_0 | 204.49 | 15.26 | **FLASH ATTENTION AT 32k: 1.22x prefill, 1.84x decode.** Without it, decode collapses from 21.68 at empty context to 9.87 at 32k — it loses 54% of its speed just for holding a long conversation. With it, the same span costs only 17% (21.91 → 18.17). ⚑ THIS IS THE OPPOSITE OF clm-0017, AND THAT IS THE POINT. That report — dense 27B, **Vulkan**, stock build — found flash attention CAUSING a prefill collapse at depth, fixed only by an unmerged contiguize patch. Its author explicitly could not test ROCm, having lost it to a Fedora 44 upgrade, and asked for exactly this replication. On the HIP path, on a stock build, **there is no cliff — FA is the thing preventing one.** Scope honestly: this is a hybrid MoE, not the dense 27B they used, so architecture is uncontrolled. A dense-model run is in flight and will settle whether the difference is the backend or the architecture. Either answer is worth reporting back. PRACTICAL: leave `-fa on`. It wins every cell measured here and wins hugely at depth. QUANTISED KV — TWO FINDINGS, BOTH NEGATIVE: · **q8_0 costs 16% decode at 32k** (15.26 vs 18.17). At empty context the two are identical (21.63 vs 21.91), so the penalty is depth-dependent and would be invisible to a shallow benchmark. · **q8_0 with flash attention OFF cannot create a context at all** — llama-bench exits with "failed to create context". Quantised KV *requires* FA on this build. Recorded as a failed cell against its fingerprint rather than retried into a pass (protocol §9). ⚑ CORRECTED 2026-08-08 (operator catch). I first wrote that these results "do not support clm-0006". That is backwards. clm-0006's claim is that **stock llama.cpp dequantizes the KV cache to full precision repeatedly during inference on this silicon**, and that a community FIX then makes q8_0 run 23-53% FASTER than f16. **We are on a stock build.** So measuring q8_0 as *slower* is exactly what clm-0006 predicts for stock — it CONFIRMS the diagnosis and says nothing at all about the fixed build, which we have not tested. That inverts what this result means. Rather than deflating clm-0006, it raises the value of testing the patch: stock costs 16% decode at 32k, the fix claims +23-53%, so the available swing at depth is roughly 40-70%. **That is potentially larger than MTP's 1.45x**, and it compounds with it rather than competing. AND IT IMPLICATES PRODUCTION. `cfg-0002` ran **q8_0/q8_0 KV at 200,000 context** on a fork carrying only the `.ckpt` sidecar fix (`con-0001`) — nothing touching KV. So production sat on the slow dequantisation path, at the deep end where clm-0006 says the penalty is largest. Together with the `--spec-draft-n-max 6` finding above, that is two independent production settings measurably below optimum. q8_0 remains a memory-saving lever on a stock build, not a speed one — but on a patched build that may reverse entirely, and it is now the highest-value untested experiment. SCATTER: every surviving cell reproduced within **1.3% CV**, most under 1%. Per clm-0017 that signature indicates a healthy code path; the broken Vulkan path scattered 5-10%. Our sweep now reports CV per cell precisely so this is checked rather than assumed.