clm-0014

On gfx1151, Vulkan measured ~22-24% faster than ROCm on the same 35B-A3B model and build — pp4096 1039 vs 840 t/s, tg128 53.1 vs 43.4 — but the Vulkan build produced GARBAGE OUTPUT for that model, so the numbers describe a broken configuration.

Measured on a Framework Desktop (Strix Halo, Q8_0, 35.21 GiB) with llama-bench. The reporter states plainly that the current llama.cpp Vulkan build returns a single garbage Chinese character when actually asked anything with Qwen3.6. ⚠ THE METHODOLOGICAL POINT MATTERS MORE THAN THE NUMBERS. **llama-bench measures throughput and never checks that the output is valid.** A backend can be 24% "faster" while emitting nothing usable, and the benchmark cannot tell. Our Tier 1 is built on llama-bench, so we inherit that blind spot exactly. Fix adopted: an OUTPUT SANITY GATE before any throughput number is recorded — one real generation per (model, backend, build), checked for coherent text, and the throughput discarded if it fails. Cheap, and it is the difference between "Vulkan is faster" and "Vulkan is broken but quick about it". SECONDARY, AND DIRECTLY USEFUL: the same runs sweep ubatch. Prefill peaks around **ub 1024-2048** on both backends (ROCm 715 -> 833 t/s from 512 to 2048; Vulkan 964 -> 1039 at 1024) and DEGRADES at 4096. Decode is flat across ubatch, as expected. That narrows our own planned sweep to a sensible range rather than guessing.

Note — the record's own working

Cited by — computed at build time, never stored