clm-0037

Qwen3.5-122B-A10B UD-Q4_K_M, f16 KV, `-fa on`, `--parallel 1`, thinking off, stock llama.cpp 3653e6d asserted at launch. τ² airline, `--max-concurrency 1`, `--max-steps 200` (tau2's default). aihydra, ROCm 7.1.0. ## The number | | value | |---|---| | mean reward | **0.545** +/-0.208 (95%) | | n | **22** — the arm hit its 6h bound at 360 min, incomplete of 50 | | split | 12 pass / 10 fail | | cut at max_steps | **0** | | turns | median 22, p90 30, max 36 | ## What it corrects Every 5-task run of this configuration returned **1.00**, and I quoted that as the model's score repeatedly — including as the reference point the whole KV-quality comparison was built on. At n=22 it is **0.545**. The first five tasks are simply the easy ones, so the 5-task figure was not a noisy estimate of 1.00; it was a confident measurement of the wrong thing. This is the concrete cost of the sampling error `clm-0036` identified, and it is larger than that claim predicted. clm-0036 argued the 5-task harness could not RESOLVE differences of ~0.4; this shows it was also BIASED, because the truncated set was not a random sample of the domain. ## Two findings that only appear at a realistic step budget **Nothing was cut at max_steps (0 of 22), and the longest task ran 36 turns.** So the 200-step budget is comfortably adequate, and my old `--max-steps 40` sat just above the observed maximum — close enough that it clipped the tail while looking generous. **The slow tasks are slow in tokens, not turns.** Two tasks in this arm each took over an hour while the GPU stayed pinned at 99%. The cause is generation length: ~10,000 tokens in a single turn, which at 19.5 t/s is ~8.5 minutes for one response. A task of 30 such turns takes hours without ever looping or stalling. ⚑ I misread this at the time and called it a retry loop, on a flat `n_tokens` field whose semantics I had not checked. The prompt-eval counts (41-397 tokens per request) showed healthy KV cache reuse of an advancing conversation the whole time. Corrected before it reached the record, but it was one step from being written down. ## Scope — read this before quoting 0.545 · **n=22 is a CONTIGUOUS PREFIX, not a random sample.** The arm ran tasks in order and was cut off by the time bound, so if task order correlates with anything, this is biased — in an unknown direction. A shuffled or completed run is needed before 0.545 is a property of the model rather than of tasks 0-21. · +/-0.208 is still wide. It excludes 1.00 decisively, which is the point; it does not pin the value. · Airline only, self-play, deterministic scoring, read paths only.