HomeQuestions › Is thinking mode worth it?

Is thinking mode worth it?

problem
Many local models can “think” before answering — emit a hidden chain of reasoning tokens ahead of the visible reply. It costs time and tokens on every request, and whether it buys anything on real multi-turn work is usually asserted rather than measured.
claim
Thinking mode is commonly treated as a global switch: on for quality, off for speed. If that were true, one setting would suit a whole fleet.
conclusion

Per-model, not global. On the lab's production 122B, thinking is a net negative: OFF solves 5/5 airline tasks where ON solves 3/5 and deadlocks indefinitely on the rest, with identical reward on every task both arms complete clm-0033. A cross-model matrix suggested thinking helps other models — but those score gaps fell inside the harness's own noise and were retracted clm-0035 clm-0036. What survives is categorical: the lever flips behaviour per model, so it has to be measured per model.

The core measurement is a clean A/B on agentic work (multi-turn jobs where the model calls tools and acts, rather than answering once): same model, same tasks, thinking gated server-side and verified per arm clm-0033. Off, the model solves every task at reward 1.0 in ~10 minutes. On, it argues in circles on the two adversarial-pressure tasks — the ones where the right move is to refuse or escalate — and never terminates, running 4h15m and 2h10m before being cut. Where both arms complete a task, reward is identical: thinking produced fewer, more expensive turns, and no measurable accuracy.

The first write-up generalised that result, and the record corrects it in public. A four-model matrix pointed the other way for every other model tested — then the noise audit landed: the identical configuration scored 0.60, 1.00 and 0.60 on repeat, a 0.40 spread from noise alone, the same size as nearly every gap in the matrix, and the per-model rankings were retracted clm-0035 clm-0036. The categorical outcomes survive — the 122B's thinking-on deadlocks reproduced across four runs — while the score-based case that thinking helps the others remains suggestive, unproven.

The operational rule this leaves is unglamorous: treat thinking as a per-model lever that is never assumed safe and never set globally. The 122B's corrected baseline, thinking off, is 0.545 ±0.208 at n=22 under the full step budget clm-0037 — decisively below the 1.00 its easy-task subset had reported, which is its own lesson in sampling. A mechanism for the deadlocks — extended reasoning giving the model more room to rationalise continuing instead of stopping — is plausible and explicitly unestablished clm-0033; what is measured is that the behaviour flips cleanly with the lever.

What this rests on

generated from the citations above — each entry shows the claim's current state, so this page cannot silently rest on withdrawn evidence

clm-0033measured-herehigh ●●●verified 2026-08-09

On tau2-bench airline, thinking is a NET NEGATIVE for this model: thinking OFF solves 5/5 tasks at reward 1.0 in ~10 minutes, while thinking ON solves 3/5 and deadlocks indefinitely on the other two (4h15m and 2h10m in unbounded runs). Same reward on every task both complete. Thinking bought no measurable accuracy and caused total failure on 40% of the set.

clm-0035measured-herelow ●○○retractedverified 2026-08-09

RETRACTED: the per-model reward rankings and quantised-KV cost reported by this run do not hold — they came from 5-task tau2 arms whose ~0.40 run-to-run noise and 40-step cap bias were only characterised afterward (clm-0036), so the reward numbers below are not usable. The corrected 122B score is clm-0037's; the corrected cross-model comparison is clm-0039's. What survives is categorical, not scored: the 122B fails to terminate some tasks with thinking on, and gpt-oss fails the domain outright.

corrected by clm-0037 clm-0039 · cited here deliberately: cited as the retracted cross-model matrix — the retraction is the finding this page reports, not a defect in it

clm-0036measured-herehigh ●●●verified 2026-08-09

τ²-bench at 5 tasks cannot resolve the differences drawn from it in this project's capability matrix. The IDENTICAL stock q8_0 configuration scored 0.60, 1.00 and 0.60 across three independent runs — a 0.40 spread from noise alone. That is the same size as most gaps in the capability matrix, so those gaps are not established. Three tasks always pass and two are coin-flips, which makes the 5-task mean effectively two Bernoulli trials. The domain has 50 tasks available; five were used.

clm-0037measured-heremed ●●○verified 2026-08-10

The 122B's real τ²-bench airline score is 0.545 +/-0.208, not the 1.00 reported by 5-task runs, which sampled only the easiest tasks in the domain. At n=22 it passes 12 and fails 10. Nothing was cut at the 200-step budget, and tasks run a median of 22 turns and a maximum of 36 — so the model's long tasks are long in TOKENS per turn (~10k), not in turns.

Related