Nemotron-3-Super-120B-A12Bbenchedguard 4/4
②Verdict
A 120-billion-parameter hybrid Mamba-2 mixture-of-experts activating 12B per token — the largest active footprint in the field, with no speculation path in llama.cpp, so it decodes at its floor clm-0005. On τ² reward it is indistinguishable from the 122B; on the tasks both models get right it needs 37% more turns and 85% more wall-clock to reach the same correct answer clm-0039 — on this lab's headline metric the incumbent leads. The quant confound is resolved: on 12 identical seeded tasks, UD-Q4_K_M and UD-IQ4_XS produce identical reward on every task while IQ4_XS takes +11% more total turns clm-0047 — quantisation cost this model efficiency, never correctness. It is the flattest model with depth measured here, holding its decode almost unchanged to 32k clm-0025, which earns it a second look wherever long-context stability matters more than raw speed. Every cross-model score above remains provisional until re-run under the pinned independent simulator clm-0043 clm-0048.
③Best configuration
| model | NVIDIA-Nemotron-3-Super-120B-A12B-UD-IQ4_XS-00001-of-00003.gguf · UD-IQ4_XS |
| engine | ggml-org/llama.cpp 3653e6d · rocm · host aihydra (igpu) |
| flags | -ngl 999 -fa on -c 32768 --parallel 1 --load-mode none --jinja -rea off -ctk f16 -ctv f16 |
| template | not recorded at test time |
| tree | upstream — stock |
backfilled aged evidence — reconstructed from the archive · config record cfg-0027
every cell generated from the record at build time · throughput cells from cfg-0015 (same build 3653e6d, fa on, f16 KV) · provenance: measured-here throughout
③aDecode against context depth
④Other configurations tested — each as a delta against best
| variant | Δ decode | Δ turns (paired tasks) | Δ energy | note | records |
|---|---|---|---|---|---|
| flash attention off | -60% @131k | — | — | The shallow cells hide it entirely — the fa-off arm only collapses at depth. | clm-0028 |
| q8_0 KV · stock build | -22% @131k | — | — | No dequant-patch arm has been run on this model. | clm-0022 |
| UD-Q4_K_M · fair-quant arm | — | -10% n=12 | +83% | Reward identical task-for-task on the seeded subset — the finer quant differs only in turn efficiency and window energy, so the IQ4_XS best-config rows understate this model's speed, never its quality. | clm-0047 |
deltas computed at build time from the named runs, paired-task traces and metered windows — a hazard row shows words where a delta would mislead
⑤Open questions
- Cross-model τ² at fair quant under the pinned simulator — the re-execution spot-check showed self-play flattens what the benchmark measures, so both quant arms need pinned re-runs before the comparison is load-bearing clm-0047 clm-0048.
- Wh per correct answer — unmeasured; window energy exists for the seeded arms only eng-0061 eng-0062.
- Long-context agentic behaviour — the depth-stability finding is throughput-only; no capability run exists above the serving context clm-0025.
- The turn distribution's long tail — its longest tasks run far past the 122B's, which is what broke the earlier capped comparison clm-0039; whether the tail is the model or the simulator is unresolved clm-0048.