clm-0030

SUPERSEDED: the Pass^1 = 1.000 reported by this run came from a 3-task subsample biased toward the domain's easiest tasks — the other two of the original five never terminated and were excluded as infrastructure errors. The sustained score across a realistic sample is clm-0037's 0.545 (n=22), which is the number to cite for this model on tau2 airline. This run was nonetheless the project's first genuine capability measurement rather than a throughput number, and it proved the harness and scoring path work.

τ²-bench v1.0.1, airline domain (14 tools), aihydra, ROCm 7.1.0, stock llama.cpp 3653e6d, Qwen3.5-122B-A10B UD-Q4_K_M, `-fa on`, f16 KV, `-c 32768`, `--parallel 1`. ## Pins — changing ANY of these re-baselines the series (protocol §8) | pin | value | |---|---| | agent LLM | the local 122B | | user simulator | **the same local 122B** | | judge | **none — deterministic scoring only** | | domain | airline, 14 tools | | tasks requested | 5 | No external judge was available offline, so scoring rests on tau2's deterministic components: database state-hash comparison and tool-call trajectory match. That is the more objective half of tau2's scoring, but it is not the whole of it. ## Result | metric | value | |---|---| | **Pass^1** | **1.000** | | Read actions | 5/5 (100%) | | Write actions | **none exercised** | | DB match | ✓3 / ✗0 (100%) | | Termination | 3 normal, all user-initiated | | Infra errors | **2** | | task | reward | turns | duration | |---|---|---|---| | 0 | 1.0 | 16 | 31 min | | 1 | 1.0 | 22 | 36 min | | 2 | 1.0 | 21 | 44 min | | 3 | — | 0 | infra error | | 4 | — | 0 | infra error | The model held 16-22 turn conversations with correct tool use throughout and reached the right database state every time. Against our own stated failure mode — tool selection — that is the first direct evidence in either direction, and it is positive. ## What this does NOT establish · **Three tasks is a small sample.** Pass^1 = 1.000 over n=3 is consistent with a true pass rate anywhere from roughly 0.3 upward. It is a floor, not a score. · **Read paths only.** Write actions were never exercised — every completed task was a lookup. Mutation is where an agent can do damage, and it is untested. · **The two missing tasks are now explained, and it matters (clm-0032).** They were `infrastructure_error` in this run because tau2 defaults to THREE concurrent simulations against our deliberately single-slot server. But re-run serially they do not fail — they **never terminate**, running 4h15m and 2h10m without producing a result. They are precisely the two tasks whose user is scripted to escalate indefinitely, and on those the agent argues instead of emitting `###TRANSFER###`. **So Pass^1 = 1.000 is computed over exactly the subset the model handles well**, and the two excluded tasks are the ones testing the hardest behaviour. The headline is real but it is not the whole picture. · **Self-play.** The user simulator is the SAME model as the agent. That is tau2's default when no second model is available, but a model conversing with itself may be an easier interlocutor than a different one would be. · **Airline is 14 tools**, far below the tool counts Warden actually runs (~51) and far below where grammar or context limits bite (clm-0024). ## Cost, which matters for planning **31-44 minutes per task** on this model, and the earlier clipped run measured 110 minutes per task while sharing the box. A 5-task domain is a 3-4 hour job; the full airline suite would be far longer. Any future capability series needs that budgeted honestly rather than guessed — my first attempt at this run was killed by a 2-hour timeout I set without measuring a single task first.

Note — the record's own working

Cited by — computed at build time, never stored