aihydra — first boot

Written 2026-08-07, the evening the box was unboxed and the evening aibeast was declared dead. Ordered by irreversibility, not importance: things that can only be captured once come first, decisions come before the installs that depend on them.

Nothing here installs ROCm or a model. This gets the box up, proves it is healthy, captures the baselines that cannot be recovered later, and stops.

Phase 0 — before you press power (2 minutes)

Record the pre-boot standby draw from sensor.hardware_ai_hydra_power.

This is the number we never had for aibeast. inc-0005 was diagnosed entirely from smart-plug statistics, and its should_have_caught_it reads: “A plug-power threshold alarm — ‘AI Power fell below 5 W while a model was supposed to be resident’ — would have fired within minutes.” You cannot set that threshold without knowing what normal looks like, and you cannot measure clean standby again once the machine has a workload.

Write down: standby watts, powered off, nothing installed.

Phase 1 — first boot: BIOS before OS

Watch the plug, not the screen. We now know exactly what the three failure curves look like (inc-0005, on_site): a healthy boot ramps and settles; a wedge sits flat around 45 W with no POST; a board-level fault shows standby with a slow hiccup hum and never latches. If aihydra does anything other than ramp-and-settle, stop and compare against that incident before troubleshooting from scratch.

Once it POSTs, go into BIOS/UEFI before installing anything and record rather than change:

• BIOS/UEFI version — and whether a newer one exists. Flashing firmware is far easier now than after an OS, a kernel and a working ROCm stack are on it.
• Memory configuration as reported
• The iGPU / UMA frame-buffer setting and its current value. Note it; do not tune it yet. GTT sizing is a kernel boot parameter on this platform, so the BIOS value is the floor rather than the whole story, and tuning either before a baseline exists means the change cannot be attributed.

⚠ Do not set amd_iommu=off yet. It is worth +5-12% on the iGPU and it disables the NPU entirely — the two are mutually exclusive. That makes it a benchmark axis and an architecture decision, not a setup step.

Phase 2 — the OS decision (yours, not mine)

The hard floor is kernel ≥6.18.4 with CWSR exported, firmware that is NOT linux-firmware-20251125, and no amdgpu-dkms. Below that line ROCm reports wrong VGPR counts and crashes, and every number measured on it is noise.

aibeast ran bare-metal Ubuntu. Consistency argues for the same; the kernel floor may argue otherwise, because distributions differ in how new a kernel they ship and how willing you are to run a mainline or HWE kernel to reach the floor. Check what your chosen installer actually provides against the floor before committing — this is the decision the rest of the stack sits on, and changing it later means a rebuild.

Related and worth deciding in the same sitting: static IP. Give aihydra its own — do not reuse aibeast’s address (<aibeast-lan-ip>). aibeast may yet come back from RMA, and two machines that have both held the same address is a debugging trap you would be setting for yourself.

Phase 3 — do these before the network is the only way in

Both of these cost an incident last time by being retrofitted:

• Record the MAC address of the wired interface, and enable and persist Wake-on-LAN. When aibeast wedged on 21 July, the out-of-band path mattered more than anything on the box. Do it at build time.
• Confirm the smart plug can hard-cycle it. The plug was the recovery path on 21 July — the wedge cleared only when power was pulled. Verify that works now, deliberately, rather than discovering it under pressure.

Phase 3.5 — HARDWARE IDENTIFIED, AND A RISK TO CLEAR FIRST

aihydra is a BOSGAME BeyondMax on the Sixunited AXB35-02 board, Ryzen AI Max+ 395 w/ Radeon 8060S, 128 GB LPDDR5-8000, 2 TB Kingston NVMe, AMI BIOS 3.11 (2026). BIOS UMA on Auto resolves to a 64 GB carveout — confirmed under Windows, which reported 128 GB installed / 63.6 GB usable.

⚠ AXB35-02 is the board in ROCm issue #6182. Every HIP workload loading a sizeable model reproduced a non-recoverable Memory critical error by agent node-0 ... Reason: Memory in use. One investigator tried 14 configurations and got 14 failures; no workaround was found.

Why this is a risk to test rather than a reason to return it:

• Those failures were on AMI BIOS 1.07 (Sept 2025), and the write-up’s premise was that the vendor had shipped no updates since launch. This unit ships 3.11, dated 2026 — a different firmware generation, and proof the premise no longer holds.
• GMKtec’s EVO-X2 uses the same physical board and does not trigger the bug (BIOS v1.12), as do Framework Desktop and Beelink GTR9 Pro on the same silicon. The fault is firmware-attributed — specifically to UMA carveout, GTT and HSA queue handling — not inherent to the board.

So the ROCm smoke test comes FIRST, before preflight tuning, baselines or anything else. If it fails, every downstream decision is moot, and you want that answer while a return is still possible.

# after the OS + ROCm are in, before ANY tuning:
#   load a model in the 60-70 GB class and watch for the fault

🔴 Test with a LARGE model — this is the part that is easy to get wrong. The fault triggers on the first sizeable GGUF upload. A 4 GB test model can load perfectly and give false confidence. Use something in the 60-70 GB class, which is where you will actually operate. On a 900 Mb line that is ~10 minutes of download to buy certainty.

Know the return window date. Not to use it — to know it. A firmware bug a vendor has not fixed is not something you can engineer around, and that deadline is the only thing in this whole plan that expires.

Phase 4 — verify the floor, then stop

./bench/preflight.sh

It checks kernel version, CWSR, firmware package and amdgpu-dkms — and is gated to Linux, so it will refuse rather than false-pass. If preflight fails, fix it before installing ROCm. Everything downstream assumes this floor holds.

Then capture the second number that cannot be recovered later:

Idle power with the OS up and no workload resident. ✅ Captured 2026-08-08: 10 W, against 1 W powered off. That pair is the calibration for the alarm below.

Phase 5 — the alarm aibeast never had

Build the Home Assistant automation on sensor.hardware_ai_hydra_power now, while the baselines are fresh and before anything depends on the box:

Measured baselines (2026-08-08): 1 W powered off · 10 W idle with Ubuntu up and no model resident. Loaded figure still to capture.

Two thresholds, not one — and this is the improvement the numbers bought. The original sketch was a single “power fell below X” alarm. But aibeast’s two failures had different power signatures, and telling them apart took retrospective forensics:

A single threshold would have fired for all three and told you nothing about which. Two thresholds separate “restart the server” from “the hardware is gone” at alert time, which is the whole difference between a five-minute fix and a twenty-hour silence.

inc-0005 was detected by absence — Warden failed over in 71 seconds and the warm-lane canary said nothing for roughly 20 hours. The sensor data existed the entire time. Nobody was looking at it. This automation is the whole lesson of that incident, and it takes minutes to build on day one versus never getting built at all.

✅ BUILT AND VERIFIED — 2026-08-08

Ubuntu Server 26.04 LTS, no LVM, no encryption, whole 1.9 TB as one ext4 root. Reachable as aihydra@<aihydra-lan-ip>.

Host floor — all four pass (bench/preflight.sh): kernel 7.0.0-29-generic (≥6.18.4) · CWSR exported · no amdgpu-dkms · linux-firmware 20260319, not the ROCm-breaking 20251125.

Memory layout — the important result. BIOS UMA set to 1 GB, but that ALONE made things worse: GTT defaults to half of RAM, so the GPU could reach only 62 GiB — less than the 64 GiB the factory Auto carveout gave. Two parameters are required, not one, because ttm.pages_limit is a global TTM ceiling that binds before amdgpu.gttsize:

amdgpu.gttsize=106496       # megabytes  = 104 GiB
ttm.pages_limit=27262976    # 4KiB pages = 104 GiB

Result: 1 GiB VRAM + 104 GiB GTT = 105 GiB GPU-reachable, with the OS still reporting 120 GiB available — GTT is a ceiling, not a reservation. Ling-3.0-flash at Q5_K_M (84.18 GB) now fits, which aibeast could never do.

104 GiB was chosen rather than the maximum deliberately: aibeast’s OOM panic was amdgpu SVM restore plus a GTT-pinned llama-server, unkillable, and a maxed ceiling both eases that failure and starves the page cache the disk-restore warm lane depends on. Revisit with measurement, not by default.

Reproducibility lockdown. unattended-upgrades, apt-daily.timer and apt-daily-upgrade.timer disabled; /etc/apt/apt.conf.d/99-aihydra-no-auto written so a reinstall of the package does not silently re-enable them; linux-firmware, linux-generic, linux-image-generic, linux-headers-generic all held. Kernel and firmware are fingerprint fields — an unattended bump between two runs invalidates the comparison with nothing in the data to show it.

SMART zero point — /dev/nvme0n1, Kingston OM8TAP42048K1-A00, fw P4ER3B31: percentage_used 0% · power_on_hours 1 · power_cycles 20 · media_errors 0 · available_spare 100% · idle 32 °C. Genuinely new, not refurbished. This is the zero point that makes a future wear rate measurable.

Defect found and fixed on first real use. preflight.sh crashed on pgrep -c llama-server, which prints 0 AND exits non-zero when nothing matches, so || echo 0 appended a second line and the numeric test died. Counting lines instead is immune to the exit status. The same class was fixed in grammar-ceiling.sh. Both scripts had never met a Linux host before tonight.

✅ ROCm CLEARED — 2026-08-08, the question the build hung on

ROCm 7.1.0 (Ubuntu archive, no third-party repo, amdgpu-dkms not even in the archive) runs a 73 GiB model on this board with zero faults. Issue #6182 does NOT affect this unit — BIOS 3.11 clears what 1.07 did not. rocminfo enumerates gfx1151 (40 CU), the CPU agent, and — unexpectedly — the XDNA2 NPU as aie2 / AIE-ML, a DSP agent visible at the HSA level rather than only through FastFlowLM.

First numbers, stock llama.cpp @ 3653e6d, HIP, -ngl 999 -fa 1 --mmap 0:

⚠ A REAL FAILURE FIRST, AND IT WAS NOT THE FIRMWARE BUG. The first attempt died with thousands of amdgpu: SVM mapping failed, exceeds resident system memory limit. Cause: mmap double-residency. The 73 GiB file sat in page cache while llama.cpp allocated the same tensors again — 122 GiB of demand on a 122 GiB machine, 1 GiB free. --no-mmap plus a dropped page cache fixed it completely.

This is a standing hazard on unified memory, not a one-off. On a discrete GPU the file cache and VRAM are separate pools; here they compete for the same DIMMs. Any run that mmaps a model close to half of RAM can hit it. amdgpu.no_system_mem_limit=1 exists as a kernel escape hatch and is currently N — worth knowing, not yet needed.

The box survived it. No panic, no wedge, memory released cleanly on kill. That is materially better than aibeast, which deadlocked unkillably on the adjacent failure (amdgpu SVM restore + GTT-pinned llama-server).

OPEN — decode looks low and the memory layout is the suspect. 21.87 t/s against aibeast’s recorded 36.5 and a bandwidth-derived ceiling of ~46 t/s (256 GB/s ÷ ~5.6 GB per token at 10B active). Prefill is comparable (324 vs 354); it is specifically the bandwidth-bound half that is down. Hypothesis: with a 1 GiB carveout everything is physically GTT, and GTT may carry an access penalty on this silicon. Note the application does NOT see a small pool — ggml_cuda_init reports Total VRAM: 106496 MiB (104 GiB) and treats it as one device memory space, so the layout works as intended from llama.cpp’s view. The question is whether GTT-backed pages cost something VRAM-backed ones would not. Competing explanations that must be excluded first: this run had no MTP, and aibeast’s 36.5 came from a different config (cfg-0002) whose flags and quant have not been matched. Do not treat 21.87-vs-36.5 as like-for-like until they are. If true that is a genuine fit-versus-speed tradeoff. Next experiment: sweep the UMA/GTT split and measure decode at each point — cheap, clean, and it decides the layout properly instead of by assumption.

Deliberately NOT tonight

• No ROCm, no llama.cpp, no models. Those come after the floor is verified.
• No production restore. Warden stays on cloud openai/gpt-5.5. Standing decision: the Strix Halo returns as a lab, not production, so reboots, flag changes and benchmarks stay free rather than disruptive.
• No tuning. GTT size, UMA, amd_iommu are all measured decisions with a baseline on either side. Tuning before the baseline exists destroys the ability to attribute the change — the same mistake as upgrading the kernel before capturing a pre-fix number.

Open decisions this raises

1. OS and kernel path — which distribution, and mainline/HWE or stock, to clear ≥6.18.4.
2. Patched or stock llama.cpp. Production ran our .ckpt sidecar fix (con-0001). With aibeast gone there is no cross-host comparison to break, so this is now a simpler question than the readiness checklist assumed — but build_commit is a config fingerprint either way, so it must be recorded, not defaulted.
3. Whether the NPU matters enough to forgo amd_iommu=off. Untested on our hardware; clm-0013 found the iGPU beat the NPU on both phases at 35B by roughly 4x, which weakens the case for protecting it.