Docs › overnight-2026-08-08

Overnight run — 2026-08-08

Tracked checklist for an autonomous session on aihydra. The operator asleep; full authorisation to download models, rebuild ROCm/llama.cpp, reboot, and load the box hard. Production Warden untouched (cloud openai/gpt-5.5). UMA is BIOS-only ⇒ needs the operator; GTT is a kernel parameter and is mine to sweep.

Every item ends DONE + recorded or BLOCKED + why. Nothing is left ambiguous.

Status key

[ ] not started · [~] in progress · [x] done · [!] blocked · [-] dropped (with reason)

A. Suite dry-run — find the gaps

  • preflight.sh — PASSED. Bug found+fixed (pgrep -c prints 0 and exits 1)
  • guard.sh — REWRITTEN. Failed 3/3 against a HEALTHY model. Three defects: raw /completion (no chat template), thinking left on so a reasoning model spent its whole budget in <think>, and budgets sized for a non-reasoning model. Then two more of my own: prompt sent on stdin which the heredoc already owned (model got an EMPTY prompt and replied “Hello! How can I assist you today?”), and nested quotes in an f-string. Now passes 3/3. A guard that cries wolf is worse than no guard.
  • sweep.sh — GAP FIXED. It never disabled mmap, so on unified memory the model file in page cache competed with the tensor allocation: free=0, 5 GiB swap in use, one cell unfinished after 20 min. Now defaults --load-mode none. Re-running.
  • cache-trace.sh — NOT RUN. Deprioritised behind the KV/FA/parallel findings.
  • capability-probe.sh — NOT RUN. Same reason.
  • [!] kv-quality.sh — BLOCKED: needs a FROZEN corpus, which is a suite dependency that must be chosen deliberately and versioned, not improvised at 3am. It is the right instrument for the KV-quant QUALITY question that clm-0022 leaves open.
  • grammar-ceiling.sh — TWO defects fixed, and the ceiling did not reproduce. See clm-0024: 200 tools / 752 KB accepted. Constraint is context, not grammar.
  • tool-calling.sh — smoke PASSED. LiteLLM reaches our endpoint and returns a correct native tool call. The adopted agentic path is no longer an assumption. Full tau2 domain runs not attempted — they need a judge model pinned first.
  • [-] export-tool-schemas.sh — DROPPED for tonight: tests/schemas/homeassistant.json already holds a PII-scanned 353 KB export. Re-export is a deliberate act after an HA/MCP upgrade, not routine.

B. Endpoint

  • Build/verify llama-server from the HIP build
  • Launch 122B — healthy in 47 s cold (73 GiB off NVMe ≈ 1.5 GB/s)
  • Fingerprints recorded in cfg-0006/0007 (llama.cpp 3653e6d, ROCm 7.1.0)

C. First complete benchmark

  • Guard recorded as run-0005 (kind: guard, 4/4 at depth 8000)
  • Sweep complete: 3 depths × KV f16/q8_0 × fa 0/1. 3 of 4 cells OK, 1 real failure (q8_0 KV + fa off cannot create a context — quantised KV REQUIRES flash attention).
  • Scatter clean everywhere — max 1.3% CV, most under 1%. Healthy code paths.
  • Ingested: 18 rows → cfg-0008/9/10, run-0008..run-0023. Guard WAIVED with a written reason (llama-bench serves no endpoint; KV-quant and FA are lossy/binary levers so capability cannot inherit from run-0005). Recorded as a gap, not an implied pass.
  • Committed and live — 25 runs, 10 configs, 20 claims.

D. MTP vs standard — the ratio is the deliverable

  • llama.cpp DOES support NextN/MTP: --spec-type draft-mtp. Model declares qwen35moe.nextn_predict_layers. No second download needed. Full type list: none, draft-simple, draft-eagle3, draft-mtp, draft-dflash, draft-dspark, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache — note ngram-mod is upstream now, so clm-0016’s 1.87x is directly testable.
  • [-] Second GGUF download — DROPPED, unnecessary; one model does both arms.
  • [!] NOTE: llama-bench has NO speculative option (-lm is load-mode, which I first misread as the spec type). The A/B must go through llama-server.
  • MTP = 1.45x (31.71 vs 21.97 floor). Recorded: cfg-0006/0007, run-0006/0007, with speculation_ratio DERIVED at build time. Matches aibeast’s ~1.50x.
  • CONFIRMED — it did. 21.97 without MTP, 31.71 with. The llama-bench 21.87 agrees with the server-API 21.97 to 0.5%, validating both harnesses.

E. Open question — decode below the bandwidth ceiling · MOSTLY CLOSED

  • MTP explains most of it. n-max sweep: 2→31.48, 3→31.71, 4→29.63, 6→25.67, 8→21.43. Optimum is 2-3; cfg-0002 production used 6, ~18% below optimum. At n-max 8 speculation is a NET LOSS.
  • GTT sweep: vary amdgpu.gttsize + ttm.pages_limit, measure decode at each
  • Not comparable — five axes differ (KV quant, n-max, ctx, build, sampling). Also measured: q8_0 KV vs f16 makes no difference to decode here (21.71 vs 21.97), which does not support clm-0006’s ‘quantised KV may be faster’ on this build.
  • [!] UMA sweep — BLOCKED, BIOS-only, needs the operator physically at the machine

F. Community cross-checks

  • FA cliff: FULLY ANSWERED — it is Vulkan-only. clm-0020 — on ROCm, FA is worth 1.22x prefill and 1.84x decode at 32k. That is the OPPOSITE of the Vulkan cliff. Their author explicitly could not test ROCm; this is the replication they asked for. Dense 27B done (clm-0021): stock ROCm 214.66 @32k / 153.63 @65k vs their broken Vulkan 94.5 / 29.8 and their PATCHED Vulkan 237.4 / 180.4. Our unpatched ROCm ≈ their patched Vulkan. The contiguize patch repairs a BACKEND defect, not a general one. Worth reporting back — they asked for exactly this.
  • ngram-mod measured: 1.17x with 38% CV on varied prompts, against the community’s 1.87x on 500 structured IFEval prompts. The benefit is highly prompt-dependent — a real qualification of clm-0016, not a contradiction.
  • Vulkan vs HIP — NOT RUN. clm-0021 makes it less urgent: HIP has no FA cliff, so the Vulkan comparison is now a curiosity rather than a decision input.

G. Beyond the 122B (operator: “continue if everything’s looking good”)

  • Qwen3.6-35B-A3B — measured and guarded 4/4. See clm-0023.
  • gpt-oss-120b / Nemotron-3-Super — NOT RUN, out of night.
  • [-] Ling-3.0-flash — DROPPED for tonight; needs the bailingmoe3 fork and PR #26608 is still unmerged. Unchanged from clm-0015’s recommendation to wait.
  • [~] Placement — two production mistunings found (n-max 6, and q8_0 KV on an unpatched build). Full re-optimisation needs the fork decision first.

H. Discipline

  • No unguarded performance rows
  • Backend STATED, never inferred
  • Failures recorded against their fingerprint, not silently retried
  • Power/temps watched; stop on anomaly
  • Never leave the box unbootable

Log

The night’s biggest finding was not planned. clm-0019: --parallel 4 destroys long-context retrieval on this build (needle 0/8) while --parallel 1 is perfect (8/8). Throughput is unaffected and silent — a sweep-only benchmark would have published the broken config. It is the first real hazard-class failure the guard tier has caught, and it is the entire argument for that tier existing.

Five defects found in our own tooling, all on first contact with real hardware:

  1. preflight.sh — pgrep -c prints 0 AND exits 1, so || echo 0 appended a second line
  2. grammar-ceiling.sh — same class, in a curl ... || echo 000
  3. guard.sh — used raw /completion, left thinking on, starved a reasoning model of tokens
  4. guard.sh — prompt on stdin the heredoc already owned; nested quotes in an f-string
  5. sweep.sh — no --load-mode none, so unified memory went to swap

The guard reported CLIFF DETECTED against a healthy model three separate times before it was right once. Probes now live in bench/guard-probe.py and are individually executable, because a probe you cannot run in isolation is a probe you cannot debug.

I. Added mid-session — the KV dequantisation A/B (the operator’s catch)

The operator remembered that clm-0006 is about stock llama.cpp dequantizing the KV cache repeatedly, which reframed a result I had written up backwards. Measuring q8_0 as slower on stock CONFIRMS that diagnosis rather than refuting it, and makes the patch the highest-value untested lever: stock costs 16% decode at 32k, the fix claims +23-53%.

  • Located the fix: Nathanw1014/llama.cpp@strix-halo-fa-fixes, commit 2a24abc “CUDA: dequantize KV on load in the tile FA kernel” — on OUR code path (ROCm uses the CUDA backend).
  • Did NOT build the fork wholesale — it sits on upstream #25xxx while our tree is 3653e6d, so that would confound the patch with the base version. Cherry-picked the single commit onto our exact base instead (ce7689f), 6 files, +201/-45, clean.
  • Confirmed the clm-0017 contiguize patch is NOT in this fork — different author, separate patch. The two community findings are independent, not a bundle.
  • [~] A/B running: stock vs patched, f16 AND q8_0, d=0/32768. f16 is measured on both builds as a CONTROL — the patch touches the tile FA kernel, so without it a general FA speedup could be misread as a quantised-KV win.