clm-0025

All on aihydra, ROCm 7.1.0, stock llama.cpp 3653e6d, `-fa 1`, f16 KV, `--parallel 1`, `--load-mode none`, 3 reps, page cache dropped between models. | model | active | quant | pp512 @0 | tg128 @0 | tg128 @32k | |---|---|---|---|---|---| | gpt-oss-120b | ~5.1B | UD-Q4_K_XL | 450.83 | **55.45** | 41.70 | | Qwen3.6-35B-A3B | 3B | UD-Q4_K_XL | 1079.38 | 51.01 | 42.45 | | Qwen3.5-122B-A10B | 10B | UD-Q4_K_M | 326.36 | 21.90 | 18.16 | | Nemotron-3-Super-120B-A12B | 12B | UD-IQ4_XS | 225.36 | **17.51** | 17.16 | THE BANDWIDTH MODEL, TESTED PROPERLY FOR THE FIRST TIME. Decode should scale inversely with ACTIVE parameters at ~256 GB/s. Taking ~0.56 bytes/param at 4-bit: | model | active | predicted ceiling | measured | realised | |---|---|---|---|---| | Qwen3.6-35B-A3B | 3B | ~152 t/s | 51.01 | **34%** | | gpt-oss-120b | 5.1B | ~90 t/s | 55.45 | **62%** | | Qwen3.5-122B-A10B | 10B | ~46 t/s | 21.90 | **48%** | | Nemotron-120B-A12B | 12B | ~38 t/s | 17.51 | **46%** | **The ORDER is right — more active parameters, slower decode, monotonically. The MAGNITUDE is not.** Realised efficiency varies nearly two-fold, and the smallest model is the least efficient: the 35B-A3B converts only 34% of its theoretical bandwidth into tokens against gpt-oss's 62%. Per-token overhead does not shrink with active parameters, so a very sparse model spends proportionally more time on everything that is not weight streaming. This refines clm-0023, which took 2.3x on a single pair as broad support for the model. With four points the honest statement is: **active-parameter count predicts ranking reliably and throughput poorly.** Useful for choosing which model to try; useless for predicting what it will do. ⚠ gpt-oss-120b RETURNS STALE ANSWERS — AND ONLY gpt-oss DOES. Its guard failed retrieval in a way none of the others did: · On a **virgin server**, needle at 8000 tokens returns `'chartreuse'` — the first word of `chartreuse-viper-88`. Retrieval WORKS; the model simply answers partially. That is a probe-strictness issue on our side, not a model failure. · **After any prior request**, the same probe returns `'the quick brown fox'` — verbatim the answer to the COHERENCE check that ran earlier. Reproduced 6/6 across two sequences. The server hands back a previous response. · Control questions are unaffected: "2+2" → `4`, "capital of France" → `Paris`. This is at `--parallel 1`, so it is NOT clm-0019 (multi-slot) and NOT llama.cpp #25992 (concurrent leakage). ⚑ **PROMPT CACHING RULED OUT, 2026-08-08.** I predicted `cache_prompt` prefix reuse. Tested both ways with a priming request in between: | cache_prompt | priming request | needle | needle | |---|---|---|---| | true | 'the quick brown fox' | 'the quick brown fox' | 'the quick brown fox' | | **false** | 'the quick brown fox' | **'the quick brown fox'** | **'the quick brown fox'** | Disabling the prompt cache changes nothing. The model returns the previous answer regardless. Combined with the virgin-server result (first request after start answers correctly with 'chartreuse'), the pattern is: **the FIRST request on a fresh server is correct and every subsequent request returns the first one's answer.** That is slot state not being reset between requests, not a caching optimisation misfiring. Mechanism still unidentified. gpt-oss uses the harmony chat format, which llama.cpp handles through a separate code path, and that remains the most likely locus — but it is a hypothesis, not a finding. Reproducible in three lines against a fresh server, so it is cheap for anyone to confirm and worth an upstream report once characterised. ADDITIONAL, from the 2026-08-08 matrix run: · **gpt-oss REQUIRES flash attention.** Both `fa=0` arms failed outright (f16 and q8_0), where the 122B only failed the q8_0/fa=0 combination. · **q8_0 costs 56% decode at 131k** (10.70 vs f16's 24.53) — substantially worse than the 122B's 35% at the same depth, on the same stock build. If the KV-dequant patch (clm-0022) helps proportionally, gpt-oss has more to gain from it than anything else measured. · f16 at 131k flagged **3.5% CV**, above our 3% scatter threshold. Per clm-0017 that is a defective-path signature and warrants a look rather than a shrug. CONSEQUENCE: **gpt-oss-120b's throughput rows are recorded but NOT guard-cleared.** They are honest numbers for how fast it generates and say nothing about whether it generates the right thing. On this evidence it is not a candidate for any Warden role until the staleness is understood — a model that occasionally serves a previous answer is far worse than a slow one. Nemotron-3-Super passed its guard 4/4 and is the slowest of the four at 17.51 tok/s, which its 12B active parameters predict. It is also the flattest with depth (17.51 → 17.16, a 2% loss to 32k, against 17-20% for the others) — worth a second look if long-context stability ever matters more than raw speed.