Cold start → first protocol-valid benchmark

Written 2026-08-12 during the repeatability audit, by walking the actual scripts and configs rather than from memory. Scenario: aihydra is gone — dead board, theft, fire — and a fresh Strix Halo box must produce (a) a throughput number and (b) a tau2 agentic number that are comparable to the published record, not merely plausible.

Read this next to docs/aihydra-first-boot.md (hardware bring-up, phases 0-3) and docs/benchmark-runbook.md (the day-to-day ordered list once the box works). This document is only the path from bare OS to the first two protocol-valid numbers.

⚠ The single largest gap this runbook cannot paper over: the NAS holds NO bench models. /volume1/Models/ contains the aibeast archive and nothing else — every active GGUF exists only on aihydra (~/models/). Until that is fixed, “restore from the NAS” means “re-download from Hugging Face and verify against the checksum ledger in audits/environment.lock-aihydra-2026-08-12.json”. See audit finding E1.

0. What survives a dead box, and where it is

1. Host floor (blocking — a number from a host below this floor is noise)

From bench/preflight.sh (each check is a hard gate there, run it rather than trusting this list):

• Kernel ≥ 6.18.4 with CWSR properties exported (/sys/class/kfd/kfd/topology/nodes/*/properties must contain cwsr_size/ctl_stack_size). Below the floor ROCm reports wrong VGPR counts on gfx1151. aihydra ran 7.0.0-29-generic (Ubuntu 26.04 LTS).
• No amdgpu-dkms (conflicts with in-tree amdgpu on Ryzen AI).
• linux-firmware ≠ 20251125 (that release breaks ROCm on Strix Halo). aihydra: linux-firmware-amd-graphics 20260319.git217ca6e4-0ubuntu3.
• GPU boot params (kernel cmdline, from the environment ledger): amdgpu.gttsize=106496 ttm.pages_limit=27262976 — the 104 GiB GTT ceiling every fit figure in the record (clm-0027 etc.) was measured against. A different GTT is a different fit universe; set it before measuring anything.
• Do not set amd_iommu=off without deciding about the NPU (they are mutually exclusive — first-boot doc).
• Stack versions to match (ledger has the exact strings): ROCm 7.1.0-0ubuntu6, hipcc 7.1.1+dfsg-0ubuntu1, Mesa/RADV 26.0.3-1ubuntu1 (this IS the Vulkan driver), libvulkan1 1.4.341.0-1, VBIOS 113-STRXLGEN-001.
• Passwordless sudo for echo 3 > /proc/sys/vm/drop_caches — every queue runner’s stop_server() drops page caches between arms; without it the drops silently no-op and cold arms are secretly warm.
• export LLAMA_ARG_LOAD_MODE=none in the bench environment (set at the top of every queue script). On unified memory, mmap double-counts the page cache against GTT (protocol §0; independently confirmed community-side, clm-0031). Never benchmark with mmap on this platform.

Verify with:

./bench/preflight.sh --endpoint http://127.0.0.1:8090   # host half fails fast pre-server

2. Builds (pinned commits, from bench/protocol.json builds)

git clone https://github.com/ggml-org/llama.cpp ~/src/llama.cpp
cd ~/src/llama.cpp
git checkout 3653e6d6d547ec763317d9ecd0ace334a7e21359   # fleet default "3653e6d"
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151
cmake --build build -j

(Flags read from aihydra’s actual build/CMakeCache.txt: Release, GGML_HIP=ON, AMDGPU_TARGETS=gfx1151, GGML_HIP_GRAPHS=ON, GGML_HIP_NO_VMM=ON. The clm-0046 backend-comparison build 77a9a66eb additionally sets GGML_VULKAN=ON in the same binary.)

The kvfix build (ce7689f, clm-0022) is 3653e6d plus one cherry-pick:

git remote add nathanfork https://github.com/Nathanw1014/llama.cpp && git fetch nathanfork
git cherry-pick 2a24abc   # "CUDA: dequantize KV on load in the tile FA kernel..."

⚠ The resulting commit SHA will NOT be ce7689f — that hash includes aihydra’s committer metadata and exists only in the dead box’s object store. The tree is reproducible; the fingerprint is not. Record the new local SHA, add it to protocol.json builds.known_builds with a note tying it to ce7689f’s recipe, and run the anchor pair (next step) before comparing anything against pre-rebuild numbers.

Any new binary requires an anchor pair before its numbers join a series (protocol builds.policy): the anchor workload — 122B UD-Q4_K_M, tau2 airline --task-ids 0 1 2 3 4 5 6 8 9 --seed 42 plus llama-bench -p 512 -n 128,512 — run on old and new binary in one session. On a cold start there is no “old binary”, so the anchor pair is run against the RECORD instead: same workload, and the delta against run-0008/0009 and clm-0045’s table is reported as the box-to-box calibration, not assumed zero.

Assert the binary before every stage (the queue runners do this; keep doing it):

. ~/bench/runmeta.sh    # sources protocol-check.sh
protocol_check_build ~/src/llama.cpp/build/bin/llama-server UD-Q4_K_M

3. Models (re-download + verify — the NAS will not save you)

Hugging Face refs from the config records; checksums in audits/environment.lock-aihydra-2026-08-12.json:

sha256sum -c <(jq -r '.models[] | "\(.sha256)  \(.path)"' audits/environment.lock-aihydra-2026-08-12.json)

A mismatch means HF re-uploaded the artifact — that is a different model for fingerprint purposes; record it as such, do not shrug.

4. Bench harness + instrumentation

scp -r lab-site/bench aihydra@<newbox>:~/bench && chmod +x ~/bench/*.sh

runmeta.sh now embeds an env-lock@1 snapshot (kernel/ROCm/Mesa/firmware/VBIOS/boot params) in every ~/run-meta.jsonl line automatically — first run_begin on the new box records the new environment with no extra step. Capture the full ledger once:

./bench/env-lock.sh > env-lock-$(hostname)-$(date +%F).json          # cheap fields, seconds
./bench/env-lock.sh --checksums ~/models > ...                       # + sha256s, minutes, nice/ionice'd

Energy prerequisites (before the first run, not after):

• The box must be on its smart plug, exposed in Home Assistant as sensor.hardware_ai_hydra_energy (cumulative kWh counter) and sensor.hardware_ai_hydra_power. Energy is always read as a counter difference across the run’s recorded started_at/ended_at window (eng records, window_provenance: recorded).
• ⚠ HA keeps 10-second raw history for only ~10 days; after that windows degrade to hourly forever. Extract energy for a run within days, not months.
• Re-measure the idle baseline for the new box before quoting any delta figure — aihydra’s baseline_w: 10.1 (box idle) and 13.6 W (122B resident-quiet) are properties of that unit (content/idle-baselines/idle-aihydra-2026-08.yaml); a new box gets a new idle-baseline record.
• Tariff: tariff_p_per_kwh comes from the observed Grid Import Price at run time (30.3 p/kWh in the existing records), units Wh/kWh/pence only — the schema rejects joules.

tau2 (agentic) prerequisites:

git clone https://github.com/sierra-research/tau2-bench ~/src/tau2-bench
cd ~/src/tau2-bench && git checkout 668d3bcd135c02aa3438f987ef45735b7c163ee3  # v1.0.1+26, what aihydra ran
uv sync    # Python 3.12 venv; the harness is invoked as `uv run tau2 ...`

• The simulator pin (protocol tau2.user_llm_pinned): --user-llm openrouter/anthropic/claude-haiku-4.5, temperature 0. This is enforced two ways and both must be restored: (1) protocol_check_tau2 refuses a different --user-llm; (2) routing is pinned account-side — the OpenRouter “EZAxis Provider Limit” guardrail allows only Anthropic as provider (first probe in the record leaked to Bedrock before this was set). A fresh OpenRouter key goes in ~/.openrouter_key; re-verify the provider guardrail with one probe before trusting a series.
• The agent side is served locally: OPENAI_API_KEY=sk-local OPENAI_BASE_URL=http://127.0.0.1:8090/v1.

5. First protocol-valid THROUGHPUT number

Order, per docs/benchmark-runbook.md steps 0-2, all against a raw llama-server on :8090, never a proxy:

./bench/preflight.sh --endpoint http://127.0.0.1:8090        # host + template gates; exit 0 required
./bench/guard.sh --endpoint http://127.0.0.1:8090 --depth 8000 --parallel 1   # 4/4 or stop
./bench/sweep.sh --model ~/models/qwen35-122b/UD-Q4_K_M/Qwen3.5-122B-A10B-UD-Q4_K_M-00001-of-00003.gguf \
  --backend rocm --depths 0,4096,32768 --kv f16 --ub 512 --fa 1 --reps 3
node scripts/ingest-llama-bench.mjs bench/results/<ts>/<cell>.json --host <newbox> --backend rocm --out content

Validity checklist for that first number: guard recorded as its own kind: guard run and referenced (or waived in words) by the performance run; --parallel 1 (llama.cpp #25992); --load-mode none; scatter ≤3% or the cell is a finding, not a result; comparison against aihydra numbers goes through the anchor-pair calibration, never raw. Expected sanity anchor: run-0009’s 21.68 tg128 at depth 0 on aihydra.

6. First protocol-valid TAU2 number

The exact shape that produced clm-0045..0049 (from bench/queues/queue-h-xmodel-series.sh, line 239 — reuse the runner rather than retyping):

~/src/llama.cpp/build/bin/llama-server -m <gguf> -ngl 999 -fa on -c 32768 --parallel 1 \
  --load-mode none --host 127.0.0.1 --port 8090 --jinja -rea off &

cd ~/src/tau2-bench
OPENAI_API_KEY=sk-local OPENAI_BASE_URL=http://127.0.0.1:8090/v1 \
OPENROUTER_API_KEY="$(cat ~/.openrouter_key)" \
uv run tau2 run --domain airline --agent-llm openai/<served-model-id> \
  --user-llm openrouter/anthropic/claude-haiku-4.5 \
  --task-ids 0 1 2 3 4 5 6 8 9 10 11 12 --seed 42 --num-trials 2 \
  --max-concurrency 1 --max-steps 200 --save-to <label>

protocol_check_tau2 on the full command line first (the queue runners do); wrap the whole thing in run_begin/run_end so the window, environment and any override land in run-meta.jsonl. Reward is the gate, turns and time-to-correct are the headline (protocol headline_metrics), and energy comes from the recorded window against the new idle baseline.

Afterwards: ingest as run + eng records the same week (the 10-day recorder horizon), and update content/ — the audit found the tau2 series of 2026-08-10/11 was never ingested as run records, which is exactly the gap that makes a dead box expensive.