clm-0039

τ² airline, `--max-steps 200` (tau2 default), `--max-concurrency 1`, f16 KV, `-fa on`, thinking off, stock llama.cpp 3653e6d. Both arms bound-limited at 6h. ## The two arms, and the distributions underneath them | | mean | n | pass/fail | turns median | p90 | **max** | |---|---|---|---|---|---|---| | Qwen3.5-122B-A10B | 0.545 +/-0.208 | 22 | 12/10 | 22 | 30 | **36** | | Nemotron-3-Super | 0.625 +/-0.237 | 16 | 10/6 | 28 | 60 | **84** | Nothing was cut at max_steps in either arm, so 200 is genuinely adequate for both. ## ⚑ Why the old 40-step budget was worse than "noisy" `clm-0036` established that my `--max-steps 40` was chosen for wall-clock convenience and clipped tasks at 41-42 turns. What that claim treated as a source of NOISE is, with the distributions now visible, a source of BIAS: · The 122B's longest task takes **36** turns. A 40-step cap almost never binds. · Nemotron's p90 is **60** and its longest is **84**. A 40-step cap truncates something like a third of its tasks. So the same bound was nearly free for one model and punitive for the other. Every cross-model number in `clm-0035` was produced under it. That is why Nemotron appeared to score 0.60 against the 122B's apparent 1.00 — the comparison was measuring *how many turns a model takes to reach an answer* as much as whether it reaches one. This is the third instance of the same error class, and the most damaging: the 2-hour tau2 timeout, the 90 °C thermal kill, and now this. The first two cost time. This one produced a wrong ranking that I published and reasoned from. ## What the corrected comparison says — ON REWARD **Nothing separates these two models on reward.** 0.545 +/-0.208 against 0.625 +/-0.237 — intervals that overlap across most of their range. ## ⚑ BUT REWARD IS THE WRONG HEADLINE, AND THE PROJECT ALREADY KNEW THAT The operator's framing, 2026-08-10: *"a model that takes more turns to get something correct is likely to be a greater frustration than one that is slower, but arrives at the correct outcome with less back-and-forth."* That is not a preference — it restates this project's OWN founding metric. `docs/14-model-backend-benchmark.md` (June 2026) sets the headline as **"time-to-correct- result + loops-to-done, with tool-call success as a gate"**. I drifted to raw τ² reward and spent this session treating turn count as a nuisance variable to be controlled for, when the original plan had it as a primary outcome. Restricted to tasks each model actually got RIGHT — turns spent failing are a different question — the two are not close: | on successful tasks | turns median | turns mean | minutes median | minutes mean | worst | |---|---|---|---|---|---| | Qwen3.5-122B-A10B | **19** | 18.8 | **2.0** | 2.4 | 6.0 min | | Nemotron-3-Super | 26 | 28.0 | 3.7 | 4.9 | **14.2 min** | Nemotron needs **37% more turns** and **85% more wall-clock** to reach the same correct answer, and its worst successful case takes **2.4x longer**. On the metric that describes what using the thing feels like, the 122B wins clearly. So the summary goes: clm-0035 said Nemotron was the standout (wrong — an artefact of the 40-step cap). Earlier in this claim I said they were indistinguishable (true of reward, and the wrong metric). **The 122B is materially better at getting to a correct answer with less back-and-forth.** One incidental finding from the same cut: failed tasks run LONGER than successful ones in both models — 122B 26 turns against 19, Nemotron 29 against 26. Failure is preceded by flailing, not by giving up early. That suggests turn count could serve as a live early-warning signal for a task that is going wrong, which is exactly the supervision hook clm-0034 identified as missing. ## Scope · Both arms incomplete — 22 and 16 of 50 — because the 6h bound fired. Contiguous prefixes, not random samples (see clm-0037). · The turn distributions are the robust part here. They are measured over every task in each arm and do not depend on the reward metric at all. · Airline only, self-play, deterministic scoring, read paths only. · Nemotron still runs at **IQ4_XS while the 122B runs Q4_K_M** — the quant confound from `candidates/nemotron3-super` is UNRESOLVED and applies to this comparison too. If anything it handicaps Nemotron further, which makes the equal-performance finding conservative rather than generous. · ⚠ This is a cross-model comparison, so it carries the unpinned-simulator confound documented in `clm-0043` — the user simulator was the model under test on both sides, not a fixed third party — and should be read as provisional until re-run pinned.