Home › Questions › Does KV quantisation cost quality?

Does KV quantisation cost quality?

problem
Long context is the point of running a large model on a big unified-memory box, but holding context costs memory — the cache that stores it competes with the model's own weights for the same fixed pool.
claim
Quantising the KV cache — storing the model's per-token working memory at reduced precision — roughly halves its footprint. The fear is what it costs: a model that mis-remembers details, or takes more conversation turns to finish the same job.
conclusion

Mostly no — once a one-commit engine patch is applied. The patch recovers +70.3% decode speed at production depth clm-0022, makes quantised KV the cheapest arm in electricity on fixed-length work — 42% less energy clm-0041 — and shrinks the agentic cost from +39% extra conversation turns clm-0038 to +9.3% clm-0045. The memory saving comes from quantisation itself; the patch removes most of what it used to cost.

The speed question settled first, with a control built in. A community patch dequantises the cache once on load instead of repeatedly during inference; measured against the same base build with only the patch differing, it moved q8_0 (8-bit quantised cache) decode by +18.3% at 32k and +70.3% at 204,800 tokens of depth, while leaving f16 (full-precision cache) unchanged at every depth — exactly the shape a genuine KV-specific fix should have clm-0022. Stock, quantised KV had been costing decode speed precisely where long context is the point.

The quality question went wrong before it went right, and the record keeps the wrongness visible. A five-task agentic benchmark (scored on whether the model completes multi-step, tool-using jobs, not on prose quality) first showed quantised KV costing 40% of task success. The finding was published, then retracted clm-0035: the identical configuration re-run scored 0.60, 1.00 and 0.60 — a 0.40 spread from noise alone, the same size as the gap being reported — so the five-task mean was, in effect, two coin flips clm-0036.

What replaced it is a better instrument. Paired on identical tasks, reward barely moved but q8_0 took 228 turns against f16's 164 — +39% clm-0038. Wall-metered electricity agreed from an independent direction: 9.48 Wh per correct answer against 6.81, +39% to the percentage point, on matched per-task windows clm-0042. Two instruments measuring different physical quantities agreeing that precisely is strong evidence the turns are the mechanism: quantised KV does not draw more power, it does more work.

Which cost governs depends on the workload. Fixed-token work pays for wall time, and there the patch makes q8_0 the cheapest arm measured — 85.3 Wh against f16's 89.6 at 200k context clm-0041. Emergent-length agentic work pays per turn — and re-run patched-vs-patched on identical seeded tasks, that penalty fell to +9.3% (234 turns against 214 over 11 pairs), with the multi-hour task blowups striking the f16 arm too: stochastic, not KV-caused clm-0045. What remains open is whether the residual +9.3% is a real cost or noise at n=11.

What this rests on

generated from the citations above — each entry shows the claim's current state, so this page cannot silently rest on withdrawn evidence

clm-0022measured-herehigh ●●●verified 2026-08-08

The community KV-dequantisation fix is real, large, and scales monotonically with depth: one cherry-picked commit recovers +18.3% at 32k, +55.6% at 131k and **+70.3% at 204,800 — production's own context** — while leaving f16 unchanged at every depth. Warden's local model was therefore running at **59% of its achievable decode speed** at the context it actually used.

clm-0041measured-heremed ●●○verified 2026-08-10

The KV dequant patch cuts energy 42% at 200k context — 146.1 Wh unpatched against 85.3 Wh patched for the same throughput benchmark — and patched q8_0 (85.3 Wh) beats f16 (89.6 Wh). That INVERTS clm-0040's agentic finding, and both are correct: on fixed-token throughput work quantised KV wins once patched, while on agentic work it loses because it spends extra TURNS. The workload decides, not the flag.

clm-0038measured-heremed ●●○verified 2026-08-10

Paired on identical tasks, q8_0 KV costs TURN EFFICIENCY: 228 turns against f16's 164 over the same 9 tasks, +39%, taking more turns on 6 of 9 and fewer on 1. Reward barely moves (1.000 vs 0.889, a single task) because reward is binary and coarse — turn count is the sensitive instrument and shows a consistent direction the mean hides. This also explains the wall-clock divergence: q8_0 was 76% slower on one task while decoding only 8% slower.

clm-0045measured-heremed ●●○verified 2026-08-11

The KV dequant patch removes most of quantised KV's agentic cost, not just its speed cost: on identical seeded tasks, patched q8_0 takes +9.3% more turns than patched f16 (234 vs 214 over 11 paired tasks) where the stock build cost +39% (clm-0038). Reward is near-identical (10/11 vs 11/11). Separately, the non-termination marathons that were attributed to q8_0 strike f16 too — this run's 3.3-hour blowup was on f16 while q8_0 solved the same task in 18 turns — so task blowups look stochastic, not KV-caused.

clm-0035measured-herelow ●○○retractedverified 2026-08-09

RETRACTED: the per-model reward rankings and quantised-KV cost reported by this run do not hold — they came from 5-task tau2 arms whose ~0.40 run-to-run noise and 40-step cap bias were only characterised afterward (clm-0036), so the reward numbers below are not usable. The corrected 122B score is clm-0037's; the corrected cross-model comparison is clm-0039's. What survives is categorical, not scored: the 122B fails to terminate some tasks with thinking on, and gpt-oss fails the domain outright.

corrected by clm-0037 clm-0039 · cited here deliberately: cited as the retracted quality scare this synthesis narrates — the retraction is part of the answer, not a footnote to it

clm-0036measured-herehigh ●●●verified 2026-08-09

τ²-bench at 5 tasks cannot resolve the differences drawn from it in this project's capability matrix. The IDENTICAL stock q8_0 configuration scored 0.60, 1.00 and 0.60 across three independent runs — a 0.40 spread from noise alone. That is the same size as most gaps in the capability matrix, so those gaps are not established. Three tasks always pass and two are coin-flips, which makes the 5-task mean effectively two Bernoulli trials. The domain has 50 tasks available; five were used.

clm-0042measured-heremed ●●○verified 2026-08-10

Per-task energy windows, matched to a common task set across arms, put a correct τ² answer at 6.81 Wh on the 122B with f16 KV, 9.48 Wh with q8_0, and 12.75 Wh on Nemotron. That is 0.21 p, 0.29 p and 0.39 p at 30.3 p/kWh. The q8_0 penalty is +39%, the same figure clm-0038 measured for turns, which is the mechanism. These supersede clm-0040's arm-level numbers, which were ~10x too high because ~90% of an arm's energy went on model loading and tasks that never scored.

Related