clm-0043

Every τ² arm was run with --user-llm set to the same model as --agent-llm, so cross-model comparisons changed the agent AND the user simulator together — exactly what the runbook forbids ("hold both --user-llm and the judge fixed across comparisons, or results re-baseline silently"). The 122B-vs-Nemotron comparisons are therefore confounded. The f16-vs-q8_0 comparisons are NOT, because both arms ran the same model on both sides.

**Core finding:** every τ² arm ran `--agent-llm "openai/$MODEL" --user-llm "openai/$MODEL"` — the user simulator was always the model under test, which `docs/benchmark-runbook.md` already said not to do. Comparisons where the simulator changed alongside the agent (122B-vs-Nemotron: `clm-0039`, `clm-0042`; thinking on/off: `clm-0033`, `clm-0035`) are confounded and should be read as provisional. Comparisons where the same model ran both sides in every arm (f16-vs-q8_0 KV: `clm-0038`, `clm-0042`; the dequant-patch throughput result, `clm-0041`, which has no simulator at all) are sound — the simulator was held constant by accident rather than by design. **The fix:** pin `--user-llm` to a single independent model for all future arms and record it in the run's config fingerprint. Re-running the confounded comparisons under a pinned simulator is the only way to de-confound them. The full argument for why an unpinned simulator has no predictable bias direction, the correlated-blind-spot problem, and the eight properties a user simulator actually needs are written up as a standing rule in `docs/methodology-lessons.md` §3.

Note — the record's own working

Cited by — computed at build time, never stored