clm-0022

METHOD — single-variable attribution. Both binaries built from the SAME upstream commit (3653e6d); the only difference is one cherry-picked patch, `ce7689f` = Nathanw1014's `2a24abc` "CUDA: dequantize KV on load in the tile FA kernel, use it for quantized decode" (6 files, +201/-45). Building the fork wholesale was DELIBERATELY avoided: it sits on upstream #25xxx while our tree is 2026-08-07, which would have confounded the patch with the base version — the attribution error clm-0017's author avoided with five same-base builds. Qwen3.5-122B-A10B UD-Q4_K_M, ROCm 7.1.0, `-fa 1`, `--load-mode none`, `--parallel 1`, 3 reps, page cache dropped between arms. | build | KV | depth | pp512 | tg128 | |---|---|---|---|---| | stock | f16 | 0 | 326.36 | 21.90 | | kvfix | f16 | 0 | 319.24 | 21.88 | | stock | f16 | 32768 | 210.21 | **18.16** | | kvfix | f16 | 32768 | 204.48 | **18.16** | | stock | q8_0 | 32768 | 200.26 | **15.29** | | kvfix | q8_0 | 32768 | 211.35 | **18.09** | **THE CONTROL IS THE POINT.** f16 decode at 32k is 18.16 on BOTH builds — identical to four significant figures. The patch touches the tile flash-attention kernel, so without an f16 arm a general FA speedup could have been misread as a quantised-KV win. It moved q8_0 by 18.3% and f16 by nothing, which is exactly what a correct KV-specific fix looks like. q8_0 prefill also gains (+5.5%); at depth 0 nothing moves, so the effect is depth-dependent as the mechanism predicts. WHAT IT CHANGES OPERATIONALLY: on stock, q8_0 KV costs 16% decode at 32k and **35% at 131k** — you pay steeply increasing speed for memory, exactly where long context is the point. **With the patch that cost disappears entirely** and slightly reverses. So quantised KV becomes free: roughly half the KV footprint at no speed penalty at any depth measured. On `cfg-0002`'s 200,000-token configuration that is ~12 GB of headroom recovered for nothing, which on a fixed-memory box buys context or a second resident model — and it is the concrete enabler for the 35B + 122B co-residency in clm-0023. ⚑ DEPTH TEST RUN — AND IT CHANGES THE CONCLUSION. My first write-up said the fix reached "parity, not superiority" and that clm-0006's "23-53% faster than f16" did not reproduce. That was an artefact of testing at only 32k. Decode at **131,072**: | build | KV | tg128 | |---|---|---| | stock | f16 | 12.03 | | stock | q8_0 | **7.81** | | kvfix | f16 | 12.04 (control — unchanged) | | kvfix | q8_0 | **12.15** | **+55.6% from the patch at 131k**, and q8_0 now sits marginally ABOVE f16 — which is clm-0006's claim, reproduced. The gain scales monotonically with depth exactly as the mechanism implies (dequantize once on load versus repeatedly during inference: the more cache you touch per token, the more you save): | depth | patch gain on q8_0 | |---|---| | 0 | none | | 32,768 | +18.3% | | 131,072 | +55.6% | | **204,800** | **+70.3%** | AT PRODUCTION DEPTH (204,800 — `cfg-0002` ran n_ctx 200,000): | build | KV | tg128 | |---|---|---| | stock | q8_0 | **5.69** | | kvfix | q8_0 | **9.69** | | stock | f16 | 9.60 | | kvfix | f16 | 9.54 (control — unchanged, -0.6% is noise) | The control holds at every depth tested: the patch moves q8_0 and leaves f16 alone. At 204,800 the patched q8_0 (9.69) now **exceeds** f16 (9.54-9.60), which is clm-0006's claim in full. Their +75% to +203% was measured at 262k, deeper than anything we ran, so our +70.3% sits just below their range on a shallower test — consistent, not conflicting. ALSO ESTABLISHED, AND WE DID NOT HAVE IT BEFORE: **f16 KV FITS at 204,800.** 9.60 t/s with memory stable at 79 GiB and no swap growth. The f16 arm was expected to be the one that might OOM against the 104 GiB ceiling; it did not. That is a real fit data point of the kind clm-0001 says we lack. 🔴 **MEASURED AT PRODUCTION'S OWN CONTEXT, AND IT IS WORSE THAN THE EXTRAPOLATION.** `cfg-0002` ran **q8_0/q8_0 KV at n_ctx 200,000** on a fork carrying only the `.ckpt` sidecar — nothing touching KV. At 204,800 that configuration produces **5.69 tok/s where 9.69 was available**: Warden's local model was running at **59% of its achievable decode speed** at the context it actually used. On a 500-token reply that is 88 seconds instead of 52. Two independent, compounding mistunings are now measured in the shipped configuration: this, and `--spec-draft-n-max 6` at ~18% below the optimum of 2-3. Neither was visible without an A/B, and neither would have shown up in any throughput number taken alone — the config simply looked like the speed of the hardware. FORK COST, NOW CONCRETE: adopting this means a SECOND divergence alongside `con-0001` (the `.ckpt` sidecar). They touch different files and both would have to be carried together. `build_commit` is a fingerprint field, so a combined tree is a configuration nobody else can reproduce — the tax is real. But a **55.6% decode recovery at 131k**, rising with depth, plus ~12 GB of headroom, on a box whose entire purpose is long context, makes this the strongest case for carrying a fork that we have. SCATTER: every cell ≤0.8% CV. Healthy code paths on both builds.