Model phases and placement — a sketch

Which models need to be warm, when, and therefore where they should live. Written 2026-08-05 from the real cron schedule. Explicitly provisional: the model choices should change once benchmarks give us a fresh view of what each machine and model can actually do. The phase structure should outlive the specific models.

1. The phases, from the actual schedule

27 cron jobs: 19 need a model, 8 are off-model commands. Plus heartbeat and unscheduled interaction.

Roughly 100 model events per day (~72 heartbeats + ~28 scheduled), and actual GPU-busy time is plausibly 1-2 hours — a duty cycle around 5-10%.

The consequence that matters

Utilisation is not the constraint. Warmth and non-overlap are.

Phase B needs a lane that is warm even when idle, because the cost of being cold is a 5-8 minute prefill. Phase D needs the biggest model available but does not care if a turn takes ten minutes. These barely overlap in time — which is precisely why the deliberate/deliver split works, and why a low duty cycle does not mean we have spare capacity where it counts.

2. The placement arithmetic — and the fact that decides it

Approximate resident cost including KV at our operating contexts:

Against usable GTT: aibeast ~90 GiB (of 96), aihydra ~120 GiB (of 128).

Only the 128 GB box can hold the 122B and an interactive model at the same time. 80 + 28 = 108 GB fits in 120 GiB. It does not fit in 90 GiB.

That single fact should drive placement more than anything else, because it determines whether an escalation from the interactive model to the deep model is a local handoff or a network hop into a cold model.

3. Candidate configurations

Config 1 — plan of record (second-halo-integration-plan.md)

aibeast: 122B · aihydra: 35B reflex + coder, 122B cold spare

• ✅ No re-tuning. The 122B stays where the patched binary, slotpin config and every measured baseline already live.
• ❌ The interactive model and the deep model are never warm on the same box, so every escalation crosses the network into a model that may be cold.
• ❌ aihydra’s 128 GB is partly spent on a spare that does nothing 99% of the time.

Config 2 — heavy pair on the big box (the operator’s proposal)

aihydra: 122B + 35B co-resident · aibeast: 27B coder + a second 35B

• ✅ Escalation becomes a local handoff. Phase B → Phase E without a network hop.
• ✅ aibeast (90 GiB) comfortably holds 21 + 28 = ~49 GB of specialists, with room — a much better fit than dedicating it to one model.
• ✅ Puts the council’s cheap members on the box that cannot host the big model anyway.
• ❌ Requires re-tuning the 122B on unproven hardware. Gated on N2 passing.
• ⚠️ ~108 of ~120 GiB is tight, and our OOM history says tight is where the box dies. observed_peak_gb from the capability probe decides whether this is real or wishful.

Config 3 — phase-switched

Daytime: 35B + coder warm, deep model idle. Overnight: unload the coder, load the heavy.

• ✅ Matches the duty cycle honestly; llama-swap already does TTL-based swapping.
• ❌ Every swap evicts warm state, and warmth is the thing we spent months buying.
• Probably only worth it if Config 2’s memory maths fails.

3a. Configuration scope — the constraint that actually limits co-residency

Placement is not only “does it fit”. Some settings are one value for the whole machine, so two models wanting different values simply cannot share a box. Those are placement constraints, not tuning choices, and they need deciding before any memory arithmetic.

System-level — one choice per machine, binds every model on it

Model/process-level — can differ per co-resident llama-server

Backend binary (ROCm / Vulkan / ROCmFPX fork) · quant · -c · -ngl · -fa · -b / -ub · -ctk / -ctv · --parallel · --cache-ram · --spec-type · chat template.

Which backends can coexist

Two subtleties worth carrying into the design

1. The ngram-mod shared pool only works WITHIN one llama-server. Its multiplier comes from concurrent slots feeding each other’s drafts (clm-0010). Give each council member its own process and that benefit vanishes. So “several agents” and “several servers” are not the same decision, and the faster arrangement is the less isolated one.
2. The NPU’s niche is now narrow and specific: small models, prefill-shaped work, short outputs, latency-insensitive. At 35B it loses both phases by ~4x (clm-0013). It is a background lane, not a tier.

And all of this moves. Backend rankings flipped once already in this project (Vulkan crashed at 128K, so we chose ROCm; the community now reports Vulkan faster — on a build that emits garbage). Treat the mapping as a dated snapshot, re-derived after any rebuild.

4. What has to be measured before choosing

1. observed_peak_gb for each model at its real context — Config 2 lives or dies on whether 108 GB is the true figure or an underestimate. Static estimates have never predicted an OOM here (clm-0001).
2. Concurrency cost. Two models resident on one box share one LPDDR5X bandwidth budget, and decode is bandwidth-bound. Co-residency may mean sequential handoff with no load penalty rather than genuine parallelism — which is fine, but changes what a council can be.
3. Whether the 122B still behaves on aihydra (integration plan gate N2).
4. Can two llama-servers share one GPU cleanly? Unverified. If yes, backend A/B and co-residency get much easier; if it thrashes, Config 2 needs rethinking.
5. Whether these are even the right models. The 27B was rejected as a reflex model on speed and may be the right coder; the 35B won on conceptual quality in June against a field that has since moved. Everything above is placement logic, and placement logic survives a change of occupants.

5. What this sketch is for

Not to fix a configuration. To make the shape explicit — that Warden spends most of its life idle, needs one permanently warm fast lane, does its heavy thinking in a window where latency is free, and only occasionally needs several models at once. Any configuration that respects that shape will work; any that ignores it will feel wrong regardless of how much hardware is thrown at it.