Docs › instrumentation-gaps

Instrumentation gaps — what the design doc asks for and what exists

Written 2026-08-10 in answer to a direct question: has energy been backfilled for every run, how many were missed, and are there other metrics we’ve skipped? The honest answers are: not yet, most of them, and yes — six record types and five of seven run metrics.

This page exists because the same failure keeps recurring: the design doc specifies something, it is never built, and nobody notices until a question exposes it. Energy was the third instance today, after doc 14’s time-to-correct-result headline and the kWh-not-joules units rule.

Record types: 6 of 12 missing

§ type status what its absence costs
2.1 node ✅ built
2.2 config ✅ built
2.3 run ✅ built
2.4 energy ✅ built 2026-08-10 was absent while 99 runs were recorded
2.5 idle_baseline ✅ built 2026-08-10
2.6 workload ❌ missing no drilldown dimension — results cannot be sliced by what was being asked of the model
2.7 telemetry ❌ missing production behaviour and benchmark behaviour are not separable; Warden’s real usage is invisible
2.8 suite ❌ missing suites are named as free strings (tau2-bench-airline@v1.0.1) with nothing defining or versioning them
2.9 decision ❌ missing no browsable log of what was chosen and why — the reasoning lives in claims and commits instead
2.10 claim ✅ built carrying more weight than intended, precisely because decision is absent
2.11 contribution ✅ built
2.12 edge ❌ missing cross-host coordination unrepresented; cross_host_pair is a bare string with nothing to point at

suite is the most quietly damaging. Every run references a suite by name, and nothing anywhere says what that suite is — so a suite could change underneath a series and no record would show it. That is the same class of hazard as the chat-template drift the config schema explicitly guards against.

Run metrics: the headline one is in a single run

The design doc’s §2.3 example lists seven metrics. Across all 99 runs:

metric present in note
decode_tps 50 / 99 the one thing consistently captured
prefill_tps 46 / 99
tool_call_success 4 / 99 specified as a GATE, so its absence is not neutral
empty_arg_calls 3 / 99
ttft_ms 2 / 99
turns_before_degradation 1 / 99
time_to_correct_answer_s 1 / 99 the project’s own headline metric
energy 0 / 99 field existed from the start, never populated

⚑ time_to_correct_answer_s is what docs/14-model-backend-benchmark.md names as the headline — “time-to-correct-result + loops-to-done, with tool-call success as a gate” — and what the operator independently identified as the thing that actually determines whether a model is pleasant to use. It is populated once.

The consequence is visible in this session’s own reasoning: model comparisons were argued on mean_score for hours before turn counts and wall-clock were pulled out of the raw simulation files by hand. That analysis should have been a query against recorded metrics, not archaeology.

Energy backfill: coverage and what is unrecoverable

Nothing is populated yet. The reconstruction has been run and validated; the eng- records have not been written. Current state is 0 / 99.

What is recoverable, and at what confidence:

set runs window source confidence
Aug 8 llama-bench sweeps ~84 JSON mtimes bracket each invocation good, but one window covers several runs — the runs are rows within a JSON, so energy is shared, not per-row
Aug 8 spec-ab / capability / cache ~9 result-directory names + file mtimes good
Aug 9 τ² runs 2 simulation directory timestamps good
τ² powered arms (Aug 9-10) not yet recorded as runs logged start/end exact
June-July runs 5 pre-aihydra ⛔ unrecoverable — AI Beast’s power sensor is unavailable and its energy counter is frozen at 120.73 kWh

So roughly 95 of 99 runs are recoverable, 5 are permanently not, and the recoverable ones inherit a caveat: the window is the invocation, not the individual row.

⏳ Deadline. Raw 10-second history is retained ~10 days; long-term statistics persist indefinitely but only hourly. The Aug 8 windows expire around 2026-08-17-18, after which those runs drop to hourly resolution — useless for invocations lasting four minutes.

What was fixed today

  • energy collection built to §2.4, with build-time enforcement: delta_w must equal mean_w_active − baseline_w, a non-recorded window must explain its derivation, and joules are rejected outright.
  • idle_baseline collection built to §2.5, with aihydra’s first record. Two of the three deltas the doc asks for are marked measured: false rather than estimated — notably igpu-resident-quiet, which is what prices keep_alive, and npu-resident-quiet, which is the question an always-on NPU triage loop turns on.
  • runs gained optional started_at / ended_at.
  • bench/runmeta.sh installed and wired into all seven bench scripts via an EXIT trap, so every invocation now emits its UTC window, duration, exit code and a contention flag to ~/run-meta.jsonl. Verified on a deliberately failing run — failed runs emit too, which is the case most likely to be silently dropped.

What remains

  1. Write the eng- records and link them, before the 10-day window closes.
  2. Capture time_to_correct_answer_s and tool_call_success as standard, not incidentally.
  3. Measure igpu-resident-quiet — a model loaded and idle. It is one server start and ten minutes of doing nothing, and it unlocks the keep_alive cost question entirely.
  4. Build suite, so a suite name means something checkable.
  5. Decide whether workload, telemetry, decision and edge are still wanted, or whether the design has moved on. Four unbuilt record types may be a plan that outgrew itself rather than a backlog — but that should be an explicit decision, not a silent omission.