The KV quantisation saga

Problem — The more context an AI model can hold, the longer it can work and converse — but holding that context costs memory, and on a home machine memory is the hard limit on what a local model can do.

Claim — Storing the context at reduced precision (“quantising” it) roughly halves the memory it needs, and might even be faster. The fear is what it costs: a model that mis-remembers details, or rambles longer before finishing a job.

Conclusion — On this lab’s hardware, a one-commit patch to the inference engine turned the fear mostly false: it recovered +70% generation speed at full production context (clm-0022), made quantised context the cheapest option in electricity on fixed workloads — 42% less energy (clm-0041) — and shrank the quality cost from +39% extra conversation turns to +9% (clm-0045), with detail-recall tests passing throughout. The memory saving comes from quantisation itself; the patch gives back what it used to cost.

Quantised KV cache is a simple trade on paper: halve the bytes each token’s attention state costs, pay for it in precision. On a fixed-memory box, that trade is supposed to buy either more context or a second resident model. Whether it was actually free, on this hardware, on the model this lab actually runs, took nine claims and one retraction to settle — and the answer that came out the other end is more useful than “yes” or “no” would have been.

A speed win, with a control built in

The saga starts with a community patch: a single cherry-picked commit that dequantises the KV cache once on load, in the tile flash-attention kernel, instead of repeatedly during inference. clm-0022 measured it properly — same base commit, only the patch differing, an f16 arm run alongside every q8_0 arm as a control. The control is the point: f16 decode moved by nothing at any depth tested, while q8_0 gained 18.3% at 32k, 55.6% at 131k, and 70.3% at 204,800 — production’s own context. That shape, growing with depth and touching only the quantised arm, is exactly what a correct KV-specific fix looks like, not an artefact of a faster kernel. Stock, quantised KV had been costing up to 56% of achievable decode speed at long context. Patched, that cost disappeared and slightly reversed.

clm-0027 filled in the memory side and found less than expected. Production’s config had assumed q8_0 saved roughly 12 GiB over f16 at 200,000 tokens; the measured difference was 2 GiB, because only a minority of this hybrid model’s layers carry a full attention cache. Quantised KV was turning out to save less memory than assumed and, once patched, cost nothing in speed. The obvious next question was whether it cost anything else.

A quality scare, and a retraction

clm-0035 went looking for that cost on τ²-bench, an agentic benchmark scored by task completion rather than perplexity. On five paired airline tasks, f16 solved all five at reward 1.00; stock q8_0 solved three of five at 0.60. Quantised KV, the headline read, cost 40% of task success. That is a legible, alarming number, and it shipped as the KV-quality finding for a day.

Then the identical stock configuration was run twice more, and scored 1.00 and 0.60. clm-0036 worked out why: on a five-task set, three tasks pass every time and two sit on a knife edge, so the mean can only ever land on 0.60, 0.80 or 1.00 — effectively two coin flips, not five independent measurements. The 0.40 spread from noise alone was exactly the size of the gap the headline had reported as a finding. The KV-quality claim was retracted. What survived was categorical rather than numeric: the 122B failing to terminate certain tasks, not a scored quality delta.

Retracting a headline result is not a footnote in this lab’s process; it is the process working. A conclusion this consequential — it would have justified abandoning the patch outright — turned out to rest on a five-task harness that could not resolve a 0.40-wide gap in the first place. Catching that before it became policy is the retraction’s whole value, and the fix it produced (bigger task sets, the actual step budget the domain needs) is now standing practice rather than a one-off correction.

Turns are the real currency

The retraction left the quality question open rather than answered. clm-0038 answered it properly, by pairing: the same nine tasks, same order, only the KV type differing. Reward barely moved — 1.000 against 0.889, a single task. Turn count moved a great deal: q8_0 took 228 turns against f16’s 164, +39%, more turns on six of nine tasks and fewer on only one.

That is the finding underneath the finding. τ² reward is binary per task, so it can only register a difference once a task flips outright; turn count is continuous and moves on every task. A metric that can only step in units of one-fifth had been standing in for the one that actually responds to the lever. It also resolved an anomaly that had gone unexplained until then: one arm spent 76% longer on a task while decoding only 8% slower — not because it was slower, but because it did more work, one turn at a time.

Energy as the confirming instrument

Wall-metered energy turned out to be a second, independent way of asking the same question, and it agreed. clm-0042 measured energy on matched per-task windows — after two earlier attempts that measured the wrong thing, first at hourly resolution, then as whole-arm totals padded by model loading and tasks that never scored. Restricted to time actually spent inside a scored task: q8_0 costs 9.48 Wh per correct answer against f16’s 6.81, a 39% penalty — the same percentage, to the point, that clm-0038 measured in turns. Two different instruments, measuring different physical quantities, agreeing to that precision is strong evidence the turns are the mechanism: quantised KV does not draw more power, it does more work.

But the same lever inverts on a different workload. clm-0041 measured a fixed-token throughput benchmark, where the task length does not depend on how many turns the model takes, and found patched q8_0 the cheapest option measured — 85.3 Wh against f16’s 89.6 Wh at 200,000 tokens of context, because the patch means the job simply finishes sooner. Fixed-length work pays for wall time, and the patch cuts wall time. Emergent-length work pays for turns, and unpatched q8_0 takes more of them. Not a contradiction — a workload-dependent sign flip, and the reason a benchmark that only reported tok/s would have caught half of it and missed the rest.

The patch’s redemption, and what is still open

The obvious next run was whether the patch that fixed the speed problem also touched the turn problem, and clm-0045 ran it under tightened methodology: identical seeded tasks, both arms patched, paired throughout. The agentic turn penalty fell from stock’s +39% to +9.3%. The same run also dissolved a piece of folklore: the multi-hour non-terminating tasks that had been read as a q8_0 property struck the f16 arm just as hard, on the same seed. Task blowups look stochastic, not KV-caused.

The current state is a sentence, not a verdict: quantised KV is cheap per token and mildly costly per task, and which one governs depends on whether the workload’s length is fixed or emergent. What remains open is whether that residual 9.3% is a real cost or noise at n=11 — the same question that opened this whole saga, asked again at a smaller scale, and still needing repeats before it is a number rather than a direction.