clm-0017

Single community source, but the confidence is high because of the METHOD rather than the reputation: five binaries built from the same upstream base, changing only which patches were present, same cells on each. That is proper attribution — the variable is isolated rather than argued about. It is the standard our own §9 asks for and rarely sees in the wild. | dense 27B, f16 KV, FA on, Vulkan | pp512 @ 32k | pp512 @ 65k | |---|---|---| | stock upstream | 94.5 t/s | 29.8 t/s | | nine patches, contiguize REMOVED | 134.5 | 45.1 | | stock + contiguize and prerequisites | 237.4 | 180.4 | | all nine patches | 252.0 | 198.2 | Removing one patch from the set reinstates the collapse; adding it to an otherwise plain build removes it. Everything else in the set is worth a few percent. Baseline for scale: ~350 t/s at empty context, so stock loses >90% of prefill purely for holding a long conversation. ⚑ THE FINDING THAT CHANGES HOW WE MEASURE, not just what we build. On the broken path identical repetitions of the same deep cell varied by **5-10%**; on the fixed path they reproduce within **0.3%**. The author reports this retroactively explained a 20% standard deviation they had flagged months earlier and never understood. We currently treat `--reps 3` as a way to average noise out. That is exactly wrong when the variance is the signal. **A cell whose repetitions disagree by more than a few percent at depth should be treated as evidence of a broken code path and investigated, not reported as a clean mean with an error bar.** Actioned: sweep.sh now computes and reports per-cell coefficient of variation and flags high-scatter cells; protocol §0 carries the rule. DECODE IS UNAFFECTED — within 0.09% across all five builds. So this is purely a prefill path issue, which also means it would be invisible to any benchmark that only reports tokens/sec of generation. HOW MUCH OF THIS TRANSFERS TO US — genuinely uncertain, and the differences are large: · **Backend.** Measured on VULKAN. The author dropped ROCm entirely because Fedora 44 version locks blocked the upgrade, so they CANNOT test the HIP path. Whether an analogous collapse exists on ROCm is open. · **Architecture.** Measured on a DENSE 27B. Our models are MoE and mostly hybrid. For Ling-3.0-flash only 7 of 42 layers hold a real KV cache (clm-0015), so a flash-attention prefill defect would touch a minority of layers and the effect size should be much smaller — but "smaller" is not "zero", and we run `-fa 1`. · **Our own unexplained decay.** clm-0016 records prefill falling from 1,211 t/s to 136 t/s at ~243k depth on HIP with a 35B-A3B MoE. Different backend, different architecture, but the same SHAPE. Worth testing as a hypothesis rather than assumed related: is there an analogous contiguity problem on the HIP prefill path? IT ALSO COMPLICATES "VULKAN IS BROKEN". We hold two reports of Vulkan misbehaving on gfx1151 (clm-0014 garbage output, clm-0016's author reporting a bug that HIP avoids). This author runs Vulkan exclusively and gets good results once patched. The honest reading is that these are DIFFERENT defects — a model-specific correctness bug and a general FA prefill inefficiency — and we should stop collapsing them into "prefer HIP". FLASH ATTENTION IS NOW A LEVER, NOT A SETTING. The author's practical advice: on a patched build FA wins every cell and can be left on; on stock the old caution still applies. Our sweep.sh hardcoded `FA=1` with no way to vary it, which on a stock build means silently measuring the collapse. Actioned: `--fa` is now a sweepable argument. FORK COST. The patch is unmerged and under discussion upstream. Adopting it would make a THIRD fork alongside our #25913 .ckpt sidecar and a possible bailingmoe3 build — and these would have to be combined, not merely chosen between. Each fork is a permanent reproducibility tax: `build_commit` is a fingerprint field, so a combined tree is a configuration nobody else can reproduce. A CONTRIBUTION WE ARE UNUSUALLY WELL PLACED TO MAKE. The author explicitly asks for replication on stock builds — depth 32k+, FA explicitly on, three runs, because a single run tells you little on the broken path. We will shortly have two Strix Halo boxes and, critically, **ROCm — which the author no longer has**. The unanswered question is not "does this reproduce on Vulkan" but "does the collapse exist on the HIP path at all", and we are positioned to answer exactly that. Qwen3.6-27B is dense and available, so we can replicate the architecture rather than only approximating it with an MoE. Verified 2026-08-06 that the branch `strix-halo-fa-fixes` exists on the Nathanw1014/llama.cpp fork. Individual commit contents were NOT verified — the patch is described here as the author describes it, not as we have read it.