Readiness — what still needs to be in place

Written 2026-08-05, before aihydra arrives. Ordered by lead time and irreversibility, not by importance: the things that need buying, and the things that can only be captured once, come first.

⛔ SUPERSEDED IN PART — 2026-08-07: aibeast is not coming back on its own

First hands-on attempt failed. The board draws standby but never latches or POSTs; symptoms are consistent with a board-level short (see inc-0005, on_site). Memory is soldered, so nothing here is user-serviceable. A warranty claim is with GMKtec.

What this changes: aihydra is the benchmark host, not the second box. Every step below that begins “power on aibeast” is void until an RMA returns or replaces it. Nothing in the protocol or runbook is aibeast-specific, so the measurement programme itself is intact — but two things do not survive:

• The pre-fix baseline and the kernel-upgrade A/B. Both were premised on measuring this box before and after. They cannot transfer to different silicon.
• slotpin_metrics.jsonl — the only production MTP telemetry we have, stranded on a healthy drive inside the dead unit. Removing that drive before anything ships is the one time-critical action, and it is blocked pending GMKtec’s answer on whether removal affects the claim.

Nothing else on this page is invalidated; re-read it against aihydra.

0. Resolved 2026-08-05 (operator) — and what each one unlocks

Net effect: the measurement programme is no longer resource-constrained. What remains is sequencing and the unknowns that only the hardware can answer.

1. Buy now — lead time, and one is a hard prerequisite

1a. First task on power-up: forensics, not benchmarks (operator, 2026-08-05)

Before anything else, find out why it died on 23 July. We diagnosed the 21 July OOM panic thoroughly and never diagnosed the second failure — the box has simply been dark since. Sequence:

1. Pull the smart-plug power history for 23 July now, from HA, before travelling. The same trick that diagnosed the current state (flat 7-10 W, no inrush on cycle) works retrospectively: a clean drop to zero means an orderly shutdown, a drop to a ~45 W plateau means it wedged, and a sawtooth before either means it was crash-looping like the 21st. That is evidence obtainable today, and it shapes what to look for on arrival.
2. On power-up: journalctl --list-boots and the previous boot’s log; check for a second kdump vmcore (the 21 July panic wrote 78 GB — a second one would too, and would also explain disk pressure).
3. Retrieve ~/slotpin_metrics.jsonl before anything can rotate it.
4. Only then decide the mitigation set.

Why this ordering matters beyond curiosity: if 23 July has a different root cause from 21 July, the mitigations already applied (--ctx-checkpoints 4, desktop removed, swap 16G) may be irrelevant to it — and a patch chosen for the wrong failure is worse than none, because it buys false confidence. It also determines whether Config 2 in model-phases.md is safe: co-residency at ~108 of 120 GiB assumes the OOM class is understood.

2. Capture before changing anything — you only get one chance

⚠️ There is a sequencing trap in our own runbook. Phase 1 says fix host health before benchmarking, because “benchmarks on a sick host are noise”. True — but if we upgrade the kernel first, we permanently lose the ability to measure what the fix bought.

Resolution: take a short, clearly-labelled pre-fix baseline on aibeast before touching kernel or firmware. Not publishable as a model result — but publishable as “what the gfx1151 CWSR kernel bug costs in tokens per second”, which nobody has quantified and which is a genuine contribution.

Also capture before any change:

• ~/slotpin_metrics.jsonl — the only production MTP telemetry that exists (21-23 Jul)
• current llama-swap and slotpin configs, verbatim
• uname -r, firmware version, dpkg -l | grep amdgpu — the “before” state itself

3. Disk — likely to block Step 5 outright

The KL-divergence base file is tens of GiB (11-37 GiB reported for 8B-class vocabs; expect more from a 122B). On top of:

• model weights: 122B ~66 GB, 35B ~22 GB, 27B ~17 GB — and a second copy on aihydra
• KV snapshots: ~1.3 GB each, several per prefix
• benchmark artefacts, logs, .kld files

Check free space on both boxes before planning the KV-quality run. If it does not fit, that step needs a smaller corpus or an external disk, and better to know now.

4. Software to stand up on aihydra

5. Host hygiene, on day one

Learned the expensive way from aibeast’s nine-day outage:

• BIOS: restore-on-AC-loss = Always On, ErP / Deep Sleep DISABLED, wake-from-S5 enabled
• WoL armed and persisted (ethtool alone does not survive reboot)
• DHCP reservation, hostname, Tailscale enrolment, SSH keys
• Record the MAC address somewhere durable before it is needed

6. Decisions still open

1. Config 1 or Config 2 (model-phases.md) — decided by observed_peak_gb, not opinion.
2. Quantized-KV fix: before or after baselining? Recommendation remains after — the A/B is the publishable result and needs an unpatched baseline to exist.
3. How long does production stay on cloud, and who calls it done? Currently open-ended. While it runs, reconciliation and the morning report send iMessage, email and calendar content to OpenAI.
4. If aihydra is DOA or faulty — is there a return window, and does the benchmark plan proceed single-box in the meantime? (Most of it can.)
5. MTP-off build for the quant × MTP ratio — worth the rebuild, or absolutes only?

7. Already done

Protocol, runbook, seven bench/ scripts, the ingest pipeline with end-to-end schema validation, real MCP schema fixtures (PII-verified), record types and the publishing path, the phase/placement sketch, and the integration plan’s N0-N6 gates.