Why written constraints kept getting ignored, and what now prevents it

Written 2026-08-10, after four constraint violations in a single day. Every one was a rule this project had already written down, and every one was caught by the operator asking a question rather than by any check.

The four

The mechanism

All four constraints live in prose, in five different markdown files, and are checked only when someone reads them. I wrote each rule, then acted from memory days or weeks later. Memory does not retain “runbook line 168” — it retains the gist, and the gist of a protocol is exactly the part that omits the constraint.

The pattern is not carelessness at the moment of action. It is that the moment of action had no connection to the moment of decision.

There is a fifth variant worth naming separately, because it has a different cause: bounds I invented for convenience and then read as properties of the system — --max-steps 40, a 2-hour τ² timeout, a 90 °C thermal kill. Those were never written down anywhere, which is its own failure: an undocumented arbitrary choice is indistinguishable from a measured constraint three days later.

What actually worked, on the same day

Every constraint that held was one made executable:

• The candidate schema refuses rejected without evidence. Negative-tested with the exact reasoning I had used to drop Maple-Preview — the build refuses it.
• A 30-character floor on gate reasons caught three of my own placeholders ("Downloaded for Phase 3C.") on the first build.
• The energy schema rejects joules outright and requires delta_w == mean_w_active − baseline_w.
• The NPU lane schema refuses metrics_inherit_from on any lane: npu record.

None of those depended on remembering anything. They fired at the moment of action.

The fix: constraints as data, checked at execution

1. One machine-readable protocol. bench/protocol.json now holds the constraints that were prose spread across five files — pinned --user-llm, --max-steps 200, --max-concurrency 1, required run fields, headline metrics, allowed energy units, comparison rules. Each carries its rationale and the claim id that motivated it, so the check explains why rather than just refusing.

2. A preflight that refuses. bench/protocol-check.sh is sourced by runmeta.sh, so every bench script inherits it. Verified against the real commands:

violating command (simulator == agent)  -> exit 1   refused
compliant command                       -> exit 0   allowed
overridden with a reason                -> exit 0   allowed AND recorded

3. Overrides are decisions, not workarounds. PROTOCOL_OVERRIDE='reason' proceeds, and writes the reason into run-meta.jsonl next to the run it affected. The goal was never to make deviation impossible — sometimes a 12B simulator is exactly what you want to test. The goal is that deviation cannot happen silently, and shows up later beside the number it changed.

4. Every run emits its own window. runmeta.sh records started_at/ended_at, duration, exit code and a contention flag via an EXIT trap — including for failed runs, which are the ones most likely to vanish.

What this does not fix

Prose rules with no executable form are still unenforced: “state your denominator”, “prefer paired comparisons”, “match quantisation across compared models”. Those are judgement, and judgement is what failed today — the energy denominator took three attempts and each correction came from a question, not a check.

Two partial mitigations, neither automatic:

• Run the audit before being asked. docs/instrumentation-gaps.md exists because the operator asked what had been missed. That audit should be periodic, not reactive.
• Publish the denominator in the claim’s own text. clm-0042 now states what is divided by what in its first sentence. A claim that cannot say plainly what its number is per is not ready.

The honest summary

Four rules, four violations, four catches — all by the same person asking what was in the denominator. The checks above convert three of the four into build-time or run-time failures. The fourth, judgement about what a metric means, has no mechanical guard, and the only real defence is stating the denominator out loud before quoting the number.