#6430 [Engine] q36: --kv fp32|fp16|fp8|fp4 for the K/V cache — 262,144 tokens on one 12 GiB card

open · @Mark-1982 · 0 commentaires · Sur GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Description

## Problem

The q36 (`qwen35moe`) path stores the K/V cache as fp32 — 48.0 KiB per token in this geometry — and parses none
of `--kv`, so the flag is swallowed at the serve catch-all (`ignoring --kv int8 (not an option of this engine)`).
On a 12 GiB card the cache, not the weights, sets the ceiling: anything past 73,728 cells is refused even though
the dense weights leave room for more. This is #5069.

Related: #3678 is a draft from 2026-10-22 covering the same flag and has not moved since; this is offered as a
working alternative with measurements, not as a repost of it.

## Result

262,144 cells load on one 12 GiB card with fp4, 131,072 with fp16 or fp8, and the fp32 default is unchanged and
byte-identical. Full numbers, method and the not-run list are in
`docs/bench-results/2026-11-20-q36-kv-precision/README.md`, which this PR adds.

On Q8_0 (37.8 GB, most of the expert weights off-card) the same ladder tops out at 204,800 cells with fp8 rather
than 262,144 — the width buys context, but the quant sets the budget, so 262,144 should not be read as
quant-independent.

**Correction to the decode numbers, 2026-11-22.** An earlier revision of this body and of the docs README led
with "262,144 cells on one 12 GiB card at 72.4 tok/s". That rate, and every rate in the results table, is 400
greedy tokens from a ~300-token prompt: a large *window* with ~300 cells occupied, not a full cache. Decode at
real fill, measured since on the same box and model (3 reps per arm, fresh nonce per rep, binary md5
`2c5a884a821286b5bcf9b74fae458f32`):

| width, window | prompt tokens in the cache | tier | decode | prefill |
|---|---:|---|---:|---:|
| fp4, 262,144 | 250,460 | on | **4.6** (4.6 / 4.6 / 4.6) | 212 tok/s |
| fp4, 262,144 | 250,460 | off | **4.3** (4.4 / 4.4 / 4.2) | 212 tok/s |
| fp16, 131,072 | 126,367 | on | **6.8** (7.1 / 6.6 / 6.7) | 479 tok/s |
| fp16, 131,072 | 126,367 | off | **6.5** (6.7 / 6.3 / 6.6) | 479 tok/s |
| fp4, 262,144 (control) | 305 | on | 74.8 | 186 tok/s |
| fp16, 131,072 (control) | 305 | on | 84.3 | 187 tok/s |

So what this change buys is the **window** — which contexts load at all on a 12 GiB card — and that part is
measured correctly. It does not buy fast decode at 250k; fill costs roughly 16x. The short-fill controls
reproduce the table's rates on this binary, so the two sets of numbers differ in what was measured, not in the
build. The ceiling table, the per-token cache arithmetic and the byte-identical fp32 default are all independent
of fill and stand as published. @archadrian raised the discrepancy on #5069 against his own 3090 run; he was
right.

## Validation

- Built for sm_86 with the release CUDA line (`STRATA_ENABLE_CUDA=ON`, `STRATA_GEOMETRY=flash_next`,
  `STRATA_NATIVE_EXPERTS=ON`, CUDA 13.1): 0 errors, both binaries relinked.
- `--kv` absent and `--kv fp32` produce output md5 `625164179b2b3e4deff739db03f2cc28`, identical to the binary
  built before the change, at 71.5/75.0 tok/s against 72.3/74.8.
- Quality on logits (`--dump-full-logits`, identical ids, 8 greedy steps, fp32 as reference): fp16 0.0044 mean KL,
  fp8 0.0064, fp4 0.0115, top-1 token identical at all 8 steps for all three. Without fp4's rotation the same arm
  is 5.19 mean / 13.30 max with a different token at 6 of 8, which is why there is no unrotated mode.
- Two soaks (12 x 320 tokens): fp16 at 131,072 and fp4 at 262,144, both flat in rate, VRAM and host memory. Both
  are short-fill, like the table.
- Decode at fill: the four long-fill arms above, 3 reps each, plus the two short-fill controls on the same binary.

Not run: no HIP or SYCL build (no twin of this file), no needle test at 200k, one prompt/8 steps for the KL
figures, `--batch N` untested with a non-fp32 width, fp4's decode cost not isolated from noise. Decode at fill was
sampled at two points only (126,367 and 250,460 cells) and only for fp16 and fp4, so nothing between ~300 and
126,000 cells is measured, and the fp32, fp8, 73,728 and 98,304 rates remain short-fill only.

The card-1 expert tier used for some of the decode arms is a separate branch and a separate PR (#6425); this
branch carries only the `--kv` work and builds from `main` without it. The fill measurements above used a build
with both merged, because the tier-on arms need it; the tier-off rows show the tier is not what produces them.

Sur le site

Liens install, modèles, releases.