clm-0038

Paired on identical tasks, q8_0 KV costs TURN EFFICIENCY: 228 turns against f16's 164 over the same 9 tasks, +39%, taking more turns on 6 of 9 and fewer on 1. Reward barely moves (1.000 vs 0.889, a single task) because reward is binary and coarse — turn count is the sensitive instrument and shows a consistent direction the mean hides. This also explains the wall-clock divergence: q8_0 was 76% slower on one task while decoding only 8% slower.

Qwen3.5-122B-A10B UD-Q4_K_M, thinking off, `-fa on`, `--parallel 1`, `--max-steps 200`, stock llama.cpp 3653e6d (NO dequant patch). τ² airline. Arms differ ONLY in `-ctk/-ctv`. ## The paired table Both arms ran the same tasks in the same order, so the overlap is a true paired design rather than two independent samples. | task | f16 reward | q8_0 reward | f16 turns | q8_0 turns | Δ turns | |---|---|---|---|---|---| | 0 | 1.0 | 1.0 | 20 | 14 | **-6** | | 1 | 1.0 | 1.0 | 24 | 24 | 0 | | 2 | 1.0 | 1.0 | 29 | 40 | +11 | | 3 | 1.0 | 1.0 | 14 | 14 | 0 | | 4 | 1.0 | 1.0 | 12 | **48** | **+36** | | 5 | 1.0 | 1.0 | 15 | 22 | +7 | | 6 | 1.0 | 1.0 | 10 | 10 | 0 | | 8 | 1.0 | **0.0** | 28 | 38 | +10 | | 9 | 1.0 | 1.0 | 12 | 18 | +6 | | **total** | **1.000** | **0.889** | **164** | **228** | **+39%** | ## Why this is the first useful reading of a question I kept mismeasuring clm-0035 claimed q8_0 cost 0.40 of reward; clm-0036 retracted it once the same stock configuration produced 0.60, 1.00, 0.60 on repeat. Both were UNPAIRED comparisons of 5-task means, and 5-task means turned out to be both noisy and biased (clm-0037). Pairing removes the sampling problem entirely: same tasks, same order, one variable. And it exposes that **I was reading the wrong metric.** τ² reward is 1.0/0.0 per task, so it can only move in steps of 1/n and needs a task to flip outright before it registers anything. Turn count is continuous, moves on every task, and here shows a consistent direction — q8_0 takes more turns on 6 of 9, ties on 3, and is shorter on 1. **Turn count should be the primary instrument for KV-quality work**, with reward as the coarse confirmation. That reverses how I have been using them. ## It also explains an anomaly I had flagged but not accounted for Arm 2 spent 4.4 h on task 8 where arm 1 spent ~2.5 h, while decoding only 8% slower (18.84 vs 20.41 t/s). An 8% speed difference cannot produce a 76% runtime difference. The paired data resolves it: q8_0 took **38 turns on task 8 against f16's 28**, and 48 against 12 on task 4. The extra wall-clock is extra WORK, not slower work. ## Scope · **n=9 paired.** The reward difference is one task and means little on its own; the turn difference is the substantive signal, and even that is 9 pairs. · A sign test on 6 improvements / 1 regression / 3 ties does not reach conventional significance. The +39% aggregate is driven substantially by task 4 (12 -> 48). · **Stock build only.** Whether the dequant patch (clm-0022) removes the turn penalty along with the speed penalty is untested and is now the obvious next run. · Both arms were cut short by the 6 h bound (22 and 9 of 50), which is why the overlap is 9 rather than 50. The pairing is sound; the sample is small. · Airline only, self-play, deterministic scoring, read paths only.

Note — the record's own working

Cited by — computed at build time, never stored