Docs › benchmark-runbook

Benchmark runbook — arrive and run

The ordered list. benchmark-protocol.md says why; this says what, in what order, and what to do when it breaks. Written 2026-08-05 while the boxes were away, so that the day they boot is spent collecting rather than designing.

Everything here runs against the raw llama-server, never through llama-swap or the slotpin proxy — except the one step that deliberately measures the proxy. A multi-hour sweep through production wedges the single slot and evicts the warm lane.

Step 0 — Preflight (minutes, blocking)

./bench/preflight.sh --endpoint http://<host>:5802

Checks kernel ≥6.18.4 with CWSR exported, firmware, absence of amdgpu-dkms, quiesce state, free memory, endpoint health — and runs the preserve_thinking acceptance test. Exit 1 means do not benchmark yet.

If the tool-call probe reports empty-argument calls: stop. That is the template defect, and every agentic number collected past it measures the template rather than the model. Nothing downstream is salvageable.

Step 1 — Capability probe (Tier 0, ~1 h)

./bench/capability-probe.sh --model <gguf> --kv f16,q8_0,q4_0 --ctx 32768,131072,200000

Loads at increasing context per KV arm until it fails, stopping each arm at its first failure. Records max fit context and observed_peak_gb — the field clm-0001 says is missing, and the reason our OOM metric has never predicted an OOM. Capture peak here; it is nearly free now and expensive to retrofit.

Do this per KV-quant arm, because it also answers the f16-fit question below.

Step 1b — Rebuild llama.cpp FIRST (possibly the cheapest win available)

A community report puts the upstream decode overhaul landing between roughly b9747 and b10218 at ~1.7x on this hardware — “if you’re on an old build of anything llama.cpp-based, rebuild”. Our production binary dates from June.

That is potentially larger than MTP, larger than the ubatch sweep, and costs a compile.

Sequencing tension, decide deliberately: rebuilding invalidates comparability with every existing number (build_commit is a fingerprint field). But the existing numbers are already on a stale build, so they are a baseline for that build and nothing else. The honest move is to measure once on the current binary, rebuild, measure again — that delta is itself a publishable result and nobody has quantified it for gfx1151.

⚠ Also rebuild the patched binary (con-0001 sidecar), or the comparison confounds a version change with a fork change.

Step 1c — Capability guard (minutes, run at EVERY config)

./bench/guard.sh --endpoint http://aibeast:5802 --depth 8000 --parallel <n>

Run this before every sweep, not once at the start. It is the cheap tier that makes “best performance without breaking capability” affordable — four checks, minutes not hours:

  1. Coherence — catches a binary cliff, where a backend is faster and emits garbage.
  2. Tool call with non-empty arguments — catches the preserve_thinking/template failure, whose signature is a call that looks successful because tool_calls is populated while every argument object is empty.
  3. Needle at depth — catches a retrieval collapse. Detects a cliff, not a slope; it is not a substitute for RULER.
  4. Cross-request isolation — only at --parallel > 1, targeting llama.cpp #25992 response leakage on gfx1151 HIP. Pass --parallel matching the config under test, or this check silently does not run.

A pass means no cliff detected, not capability intact. It cannot clear a lossy lever: if this config differs from its capability baseline by weight or KV quant, run Step 5 and set capability_basis: measured. The schema rejects inheriting capability across a quant or backend change (protocol §1a), so this is enforced rather than trusted.

Record the guard as its own run (kind: guard) and point the performance run at it via guard: run-XXXX. A performance run with neither a guard nor a written waiver fails the build.

Step 2 — Throughput sweep (Tier 1, hours)

./bench/sweep.sh --model <gguf> --backend rocm \
  --depths 0,4096,32768,131072 --kv f16,q8_0 --ub 512,2048 --fa 0,1 --reps 3
node scripts/ingest-llama-bench.mjs bench/results/<ts>/<cell>.json \
  --host aibeast --backend rocm --out content

One llama-bench invocation per (KV × ubatch) cell — those are fingerprint fields and must not be mixed inside one output file. Depth is not, so depths sweep together.

Sweep --fa 0,1, do not assume it. On a stock upstream build a dense model on this silicon loses >90% of prefill at depth with flash attention on; with the unmerged contiguize patch, FA wins every cell (clm-0017). Which side of that patch our build sits on changes the correct setting, so it is a lever to measure, not a default to inherit.

Read the scatter line, not just the means. The sweep flags any cell whose repetitions disagree by more than 3%. High scatter at depth is the signature of a defective code path — treat a flagged cell as a finding to chase, not an error bar to publish.

Covers three open questions at once: the ubatch sweep we have never run, the backend × context grid, and the quant × MTP ratio (add a --kv arm and an MTP-off build to complete it — the transferable number is the ratio, not the absolute).

Step 3 — Cache capability trace (Tier 2, ~1 h) — the part nobody else publishes

./bench/cache-trace.sh --endpoint http://<host>:5802 --tokens 20000

In-slot reuse, cross-session divergence, slots, and disk save/restore. Do not shrink --tokens: measured at 857 tokens once, reuse looked like 40%; at realistic bootstrap scale it was ~97%. That artefact cost a day.

Includes the page-cache dependence measurement we have never made — drop caches between save and restore. If the cold and warm restore times match, our published ~105 ms figure was never touching disk and is wrong.

Step 4 — Served-path delta (~2 h)

llama-benchy (MIT, maintained) against the same model twice: once at the raw server, once through slotpin → llama-swap. Its --enable-prefix-caching mode separates the cold context load from cached reuse, and its output mirrors llama-bench so the two sit side by side. Publish the gap — that difference is the cost of our own architecture, and nobody else can report it. Sanity-check a few readings against the proxy’s /_stats before trusting the delta.

Step 5 — KV-quant quality (overnight) — the contribution

./bench/kv-quality.sh --model <gguf> --text <corpus.txt> --kv q8_0,q4_0 --ctx 131072

Primary: llama-perplexity --kl-divergence. Records f16-KV logits once, replays q8_0 and q4_0 against them. The script verifies the flags against your binary before starting, because flag names have moved between releases and a wrong one wastes an overnight run. Mean KLD, ΔPPL, Δp percentiles, top-1 agreement — with uncertainty bars, single forward pass, no generation. Set -c to the full context so the whole cache is exercised. Note the .kld file is large (tens of GiB).

Secondary: RULER-lite via lm-evaluation-harness (not NVIDIA/RULER — see protocol §8): ruler_vt, ruler_cwe, ruler_fwe, niah_multikey_3 at 32K and 128K, 30-50 samples. A cliff detector, not a precision instrument.

Read protocol §8 before starting. RULER at any affordable sample count resolves 5-10pp, while the published effect size is ~1pp — so it can only tell us “catastrophic or not”. And the hybrid-architecture confound may mean there is very little to measure.

Step 6 — Tool calling (~2-4 h)

./bench/tool-calling.sh --setup                          # one-off: clone + uv sync
./bench/tool-calling.sh --smoke --endpoint http://<host>:5802   # DO THIS FIRST
./bench/tool-calling.sh --domain airline --limit 10 --endpoint http://<host>:5802

Run --smoke before any real run. tau2-bench reaches models through LiteLLM, and pointing LiteLLM at a local OpenAI-compatible server uses a pattern that is not spelled out in tau2-bench’s own docs — it is the single most likely thing to fail. The smoke test checks it twice: a direct HTTP call with a real tools= array, then the same call through LiteLLM exactly as tau2-bench makes it.

τ²-bench, MIT and active, via LiteLLM against our endpoint — real native tools= calling on our own jinja template. Hold both --user-llm and the NL-assertion judge model fixed across comparisons, or results re-baseline silently.

./bench/grammar-ceiling.sh --endpoint http://<host>:5802 --max 120 [--schemas <dir>]

Binary-searches the ceiling in ~7 requests and reports it in bytes as well as count, because bytes are the actual limit. Then the probe stays bespoke because neither τ²-bench (13-17 tools) nor MCP-Bench (bypasses native tool-calling entirely) exercises it. Lift MCP-Bench’s mcp_servers/ corpus as raw schema fixtures and scale a real tools=[...] array until HTTP 400, extending the manual 2026-07-05 probe (55 = 200, 60 = 400).

Step 7 — Cross-host overlap (when both boxes are up) — run it or lose it

Same suite, same config, both machines, cross_host_pair set. Isolates hardware as the only variable. Almost nobody has this comparison because most people replace rather than overlap, and the window closes when aibeast is repurposed.

Open questions — for the operator

  1. Is the 122B’s KV cache mostly linear-attention? — PARTLY ANSWERED 2026-08-05. The Qwen3.6-35B-A3B model card gives the layout explicitly: 40 layers as 10 x (3 x Gated DeltaNet -> MoE, then 1 x Gated Attention -> MoE) — so only 10 of 40 layers (25%) hold a KV cache at all, and those use just 2 KV heads. That matches the independently reported ~1.6 GB of KV+state per 128k tokens for this family. Consequence: -ctk/-ctv touch a quarter of the layers. A near-zero quality delta would be an ARCHITECTURE result, not a quantisation one, and must be published as such. Still to confirm: the 122B’s own layer ratio, which is likely similar but unverified. Step 5 is probably worth ONE night to characterise, not a campaign.
  2. Does the f16 arm fit at 128-200K? If not, the three KV arms are not comparable at the top length and the matrix needs redesigning around what loads. Step 1 answers it.
  3. Do we adopt the community quantized-KV fix before or after baselining? Before gives the best numbers; after gives a clean A/B of the fix itself. Recommendation: after — the A/B is the publishable result, and it needs an unpatched baseline to exist.
  4. Email Minervini for harness-bench access? No public repo, no licence. Near-zero cost to ask; unknown response time. Meanwhile Step 6 stands alone.
  5. MTP-off build for the ratio? Completing the quant × MTP matrix needs one. Worth the rebuild, or accept absolutes only?