Methodology lessons

Standing rules extracted from measurement failures, not a diary of them. Each rule cites the claim(s) it was drawn from and, where a claim’s own note now only summarises, that claim’s full diagnosis is what to read for detail. Nothing here is a claim in its own right — it exists to keep the same mistakes from being made again.

1. A small-n benchmark can silently become a coin flip

A benchmark mean is only as informative as its resolution. clm-0036 found that a 5-task τ²-bench arm has exactly three tasks that pass every time and two that sit on a boundary — so the “5-task mean” can only ever land on 0.60, 0.80 or 1.00, and is effectively two Bernoulli trials, not five independent measurements. Repeating the identical configuration produced 0.60, 1.00, 0.60: a 0.40 spread from noise alone, the same size as most of the gaps the project was reporting as findings (clm-0035).

Rule: before trusting a mean over n tasks, compute the resolution it actually offers — the binomial standard error at the observed pass rate — and compare it to the effect size you intend to report. If the two are close, the harness cannot resolve the claim; run more tasks or state the result as unresolved. clm-0036 worked this out concretely for τ² airline: at n=50 the 95% interval is roughly ±0.11, enough to resolve a 0.2 gap; at n=5 it is not. The corrected, full-sample numbers this produced are clm-0037 (single-model) and clm-0039 (cross-model).

2. Sanity-check an instrument against known bounds before trusting its output

clm-0029 set out to measure KV-quantisation quality via llama-perplexity --kl-divergence and got a baseline PPL of 39.4 on wikitext — implausible on its face for a model of this class, which should score single digits — and a quantised arm that scored better than its own f16 baseline, which quantisation cannot do. Both anomalies pointed the same way: the instrument was broken, not the model.

Rule: when a measurement violates a bound you already know to be true (perplexity that can’t improve under quantisation; a baseline PPL an order of magnitude off a known range), suspect the instrument before the phenomenon. Diagnose by elimination — vary one input at a time (context size, flash attention, corpus, model) until the defect is isolated. clm-0029 did this in four controlled passes and localised the defect to llama-perplexity itself on this build, not the corpus, not flash attention, not the model. The KV-quality question was then answered a different way, through τ²-bench task completion (clm-0035, clm-0038).

3. Pin the user simulator, or cross-model comparisons are confounded

Every τ²-bench arm in this project ran --user-llm set to the same model as --agent-llm. docs/benchmark-runbook.md already said not to do this — changing the user simulator alongside the agent re-baselines the whole series — and the practice drifted anyway (clm-0043). The bias has no predictable sign (a stronger simulator makes tasks both harder and easier) and correlates the agent’s and simulator’s blind spots, which flatters every model in self-play and flatters larger models most.

A user simulator needs to be, in order of importance (clm-0043):

Pinned — identical and versioned across every comparison in a series.
Independent of the agent under test — a different model, ideally a different family.
Faithful to the task goal — holds its persona, doesn’t leak or withhold information the specification doesn’t call for.
Realistically unhelpful — vague, partial, changes its mind, rather than cooperative and well-organised (which inflates every score).
Consistent adversarial pressure where the task specifies escalation or refusal.
Terminates reliably — a simulator that won’t end a conversation destroys the run; one τ² task ran 4.4 hours and never scored (clm-0040).
Low variance, ideally seeded — a candidate explanation, untested, for the noise in rule 1: the simulator may be a noise source in its own right.
Cheap — it is scaffolding, not the work being measured (see rule 5 below).

Rule: any comparison across models that did not pin the simulator is confounded and must be labelled provisional until re-run pinned. Same-model comparisons (e.g. one model’s f16 vs q8_0 KV) are unaffected, because the simulator was held constant by construction even though it wasn’t pinned by design.

4. A bound chosen for convenience is not a property of the system

Three separate instances of the same error: a --max-steps 40 cap chosen to keep τ²-bench arms to a coffee break instead clipped tasks mid-completion and turned a continuous score into noise (clm-0036) and then into bias, once the turn distributions were visible — 122B’s longest task is 36 turns, Nemotron’s is 84, so the same 40-step cap was nearly free for one model and punitive for the other (clm-0039). The same pattern shows up in a 2-hour benchmark timeout that clipped an earlier run before a single task completed, and in a 90°C thermal-kill threshold chosen without measuring what the workload actually needed.

Rule: any bound set to control wall-clock, cost, or safety (a step cap, a timeout, a kill threshold) is a scaffolding decision, not a finding. Before trusting results produced under it, check the bound against the real distribution it constrains — did anything actually hit it, and does it bind differently across the things being compared? If you don’t know, that is the first thing to measure, before the headline number.

5. Match the energy denominator to the question

The energy-per-correct-answer figure went through three attempts before it meant what it claimed to mean (clm-0040 → clm-0042):

Hourly power statistics, differenced — too coarse for anything shorter than hours.
Whole-arm energy ÷ correct answers — arm-level totals include model loading, idle gaps, and tasks that never scored; only 8–27% of each arm’s wall time was actually inside a scored task, so this over-counted by roughly 10x and wasn’t comparable across arms that reached different, unmatched subsets of the task list.
Per-task energy windows on a task set common to every arm — removes both problems: each task’s own start_time/end_time bounds its energy, and restricting to tasks every arm completed removes the contiguous-prefix bias (clm-0037).

Even step 3 wasn’t the end of it: self-play means the measured energy includes the user simulator generating its half of the dialogue, which is scaffolding, not billable work (clm-0042). Apportioning by generation-time share (76–82% agent) widened the gaps rather than changing the ranking, and is still a lower bound — tool calls and simulator turns draw closer to idle power than the agent’s own generation does.

Rule: “Wh per X” is only meaningful once X is defined precisely enough to match the question being asked. Arm-level and task-level denominators are not interchangeable, and neither is neutral about self-play overhead. State which denominator was used, and prefer the narrowest one that still answers the question. See clm-0041 for the companion case — the same lever (quantised KV) inverts sign between a fixed-token throughput denominator and a variable-length agentic-turns denominator, so denominator choice isn’t just a precision issue, it can flip the conclusion.

Where the full diagnoses live

This page states the rules; the claims below carry the worked diagnosis each rule was drawn from, trimmed to their core finding plus a pointer back here:

clm-0029 — the llama-perplexity instrument diagnosis (rule 2).
clm-0036 — the 5-task/Bernoulli-trial statistics lesson (rule 1).
clm-0043 — the unpinned-simulator confound and the eight simulator properties (rule 3).