#6176 [Engine] q4u32: the 4-bit KV cache with a fitted step as the default on RDNA4

open · @monscoro · 0 comentários · No GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descrição

**Problem.** Above 8K context the engine keeps the KV cache at 8 bit (8.25 bit per value with the per-row
scale) — what every published number was measured with. On RDNA4 that memory is worth more in the expert
cache than in the KV cache, and the 4-bit mode Strata has (`q4_0`, #21) gives up accuracy it does not need
to give up: a block's step is derived from the block maximum, and after the Hadamard rotation the values are
roughly Gaussian, so the maximum is a poor scale.

**Resulting behavior.** Without `--kv` the engine now picks **`q4u32`** on gfx12 (RDNA4), above 8K of context
and where the matrix-core prompt attention is on by default (Windows and Linux): the same uniform 4-bit grid
after the same rotation, with the block step **least-squares fitted**
instead of taken from the maximum — 4.5 bit per cell, one step per 32 values, no codebook, no zero point, no
calibration. Everywhere else the Terminal names the format it always named (8-bit above 8K of context,
16-bit up to 8K), and a format the person named is always written, so
every start outside that one case is byte-identical to 0.2.17.

Measured on one card (AMD Radeon RX 9060 XT 16 GB, gfx1200, Windows, ROCm 7.2):

- **at 32K context**: the prompt rate is level with `int8` — 664.5 against 669.0 tok/s (9 and 6 requests;
  the paired A/B of the coupled switch gave −1.8 % mean, −2.1 % median)
- **at 106K tokens** (the needle harness's `128k`, which cuts the text by characters): the same, with the
  same number of expert-cache slots — 737.5 against 743.4 tok/s (−0.8 % mean, −0.1 % median, 3 requests
  each) — and with the cache sized automatically **3608 against 3079 slots (+17.2 %)**
- 54.5 % of `int8`'s bytes per cell (144 against 264), which the expert cache takes up as **+18.7 % more
  resident expert slots** (32K) and **+17.2 %** (106K)
- without the matrix-core prompt path the same format is 2.5× slower (265 against 655 tok/s) — the path is
  the precondition for the default, not an optimization of it
- **the format is not equal to `int8` in quality at long context** (measured on the 13.11., 131.060
  positions, teacher-forced against fp16, the same texts): `q4u32` reaches a mean KL of 0.06949 against
  `int8`'s 0.06238 — paired **+0.00530**, t **+3.44**, 95 % interval [+0.00228, +0.00832], top-1 98.31 %
  against 98.49 %. At 4K the same comparison is +0.00052 (t −0.56), that is, not resolvable. The default
  covers a range in which the format is *not* equal to int8; the loss is small, but it is measured and
  named here.

**Why this approach.** The 4-bit pool reads a block's codes and folds the block's step into the operand of
the gfx12 WMMA kernel (#329), so the format is served at the matrix-core rate instead of a scalar one. The
default is limited to the parts that have that path (gfx12, where the matrix-core prompt attention is on by
default): a part that would run the 2.5× slower route does not get the format by default. `STRATA_HIP_WMMA=0` keeps 16-bit, which
is the no-prompt-path state.

**Compatibility.** Outside the case above nothing changes:

- an engine built from this tree names itself `0.1.63-q4u32` — the numbers of the release it was changed
  from plus a variant suffix; an upstream build has no suffix and no new default
- the Terminal names the format in a run config (its own rule: 8-bit above 8K, 16-bit below). It leaves
  `--kv` out exactly when the engine is a q4u32 build on a gfx12 card and the context is above 8K; every
  other combination keeps naming the same format as before
- an engine package carries its variant (`flavor`) beside its version, and an update never replaces an installed
  `q4u32` build with an upstream one: a newer build of the same variant is offered, a build of another variant is
  not (an installation without a variant takes what the signed feed offers — that feed carries upstream builds)
- the new surface is small and explicit: `setup --kv q4u32` and `strata launch --kv q4u32` name the format,
  `setup --kv q4lm8v` names the research format with the exact-cell window (it exists only in a build from
  this tree, and the installer says so), and three setup switches serve a self-built engine: `--engine-dir`,
  `--prefer-self-build` and `--kv-check` (a diagnostic that reads a config and reports its KV settings).
  Without that opt-in the installer takes the release's ready-made engine, as before.
- the app's format picker gains `Q4U32` and an **Automatic** entry, and Automatic is its default: an install
  from the app names no format and lets the engine pick (the app used to name 8-bit itself, which kept the
  new default from running there). A format the person picks is written, as before.

**Scope.** This changes a default, so the scope is proposed in #6175; the comment there states the **complete** scope of this
pull request, so it can be agreed (or cut) before the review. In short:

- **the default**: without `--kv`, on gfx12 (RDNA4) above 8K of context, the engine picks `q4u32`; the Terminal leaves
  `--kv` out exactly in that case, every other combination names the same format as the release — and the app's
  format picker leaves the choice to the engine with its new **Automatic** default instead of naming 8-bit itself
- **the engine work the default needs**: the `q4u32` format (4.5 bit per cell, one fitted step per 32 values), the 4-bit
  pool in the matrix-core prompt path, the engine's variant suffix `0.1.63-q4u32` with the `"flavor"` in the package
  manifest (an update never replaces an installed `q4u32` build with an upstream one) — and **the matrix-core prompt
  attention becomes the default on gfx12 under Windows**, where it was opt-in before (`STRATA_HIP_WMMA=0` turns it off)
- **opt-in additions** (explicit flags only): `setup --kv q4u32` and `strata launch --kv q4u32`, the research formats
  behind their own engine flags (`q4lm32`, `q4lm8*`, `q4u32z`) with the exact-cell window, `setup --kv q4lm8v` (a format
  that exists only in a build from this tree), and three switches for a self-built engine: `--engine-dir`,
  `--prefer-self-build`, `--kv-check`
- **the instruments**: the parity tests, the measurement tools under `tools/`, the record with its pre-registrations and
  the two papers — no behaviour, no default

Predecessors and related work: #21 (the `q4_0` mode and the Hadamard rotation), #329 (the gfx12 WMMA prompt kernel), #5535
(the same question on RDNA3, open).

**Validation.** Revision `67c292c66` on upstream 0.2.17, Windows 11, Python 3.12, AMD Radeon RX 9060 XT (gfx1200), ROCm 7.2.

Passed:

- `hip_prompt_attn_wmma.exe 32768 512 3` (HIP, built from this revision): `FAILURES: 0`, exit 0, and all
  nine `NEWHASH` values identical to the ones measured before the rebase onto this base (`232d4781a2b59ccc`,
  `2455dbed6eba900f`, `9f901d5231c4013e`, `d791e4bfb2fd064d`, `82ef1dc3fa5dd4e7`, `6c01e0a92922cca5`,
  `ad466135a4d13e70`, `06c917251ed1d5cf`, `1d3b59ff22ef3ea5`) — the merge changed no arithmetic
- **the 106K series (A.28), end to end on the card with this revision's engine** (`0.1.63-q4u32`, SHA-256
  `f2fd770c…`): four arms, 12 of 12 needles found, VRAM budget 15.419 MiB in every arm, no paging warning,
  and in every arm the route line of the **default** — `the prompt attention on matrix cores on (gfx1200) by
  default; STRATA_HIP_WMMA=0 turns it off` — with the 4-bit route line in both `q4u32` arms and in no `int8`
  arm
- **the engine's own default with its context gate** (A.28, addendum): two starts on the card with no `--kv`
  — `--max-context 8192` prints `KV storage: fp16 (the default up to 8K of context; --kv names another)`,
  `--max-context 16384` prints `KV storage: q4u32 (the default on this part above 8K: …)`. A start with no
  `--kv` and 8K of context or less is therefore byte-identical to the release, even at the engine itself.
- `python -m pytest terminal/tests/terminal`: 3612 passed, 160 skipped, 1647 subtests passed
- `python tools/test_setup_<name>.py` for every `terminal/tools/test_setup_*.py` (76 files), and
  `python -m unittest tools.test_setup_download_length tools.test_setup_local_engine` from `terminal/`: all
  green (`test_setup_download_length` has to run as a module)
- `npm run typecheck` and the app's tests of the changed files (`native-fit`, `ui-do`, `kv-choice-family`,
  `native-installed-kv`): 61 passed

Failed:

- `python -m pytest terminal/tests/terminal`: 12 failed. All twelve are environment cases of this machine
  and reproduce on the unmodified base: no `bash` (`test_docker_files`, `test_launch_install`,
  `test_shell_completion_6086`), no symlink right for the fixtures (`test_bench`, `test_monitor_bundle`,
  `test_recipes`), no SYCL toolchain (`test_sycl_build_script` ×2), a German Windows error message
  (`test_update_bat_stub`), a restricted ACL (`test_private_files`), and a real `Strata-data` folder beside
  the checkout, which the SYCL path fallback finds (`test_drafting_sycl_paths` ×2)
- `npm test` (app/host): 5 failed, all environment cases of this machine — two symlink tests
  (`chat-shell`, `model-directory`; `EPERM`, this Windows has no symlink right), two writes into `%TEMP%`
  (`agent-tools-settings`, `web-ask`; `EPERM` on rename) and one that passes when run alone
  (`agent-plan`). None of them reads a file this change touches.

Not run:

- **CUDA and SYCL builds.** No CUDA toolkit and no oneAPI on this machine. This change touches
  `engine/src/kernels/cuda/*.cu` (one source tree for CUDA and HIP) and
  `engine/sycl/src/kernels/cuda/*.dp.cpp`; both are
  unverified by compilation.
- **The RDNA3 and APU arms** (gfx11, Strix Halo / 780M): pre-registered in the measurement record, not
  measured — no such card in this machine (#5535 is the RDNA3 question).
- **The 32K rates as such.** They come from the campaign before the rebase onto this base (engine
  `0.1.62-q4u32`); the parity gate on this revision shows that the arithmetic is unchanged (identical
  checksums), and the 106K series above was measured with this revision's engine (`0.1.63-q4u32`).

**Limits.** The default is a measured change on one card family and does not reach other parts; the rule
"change a default only where you measured it" is satisfied for gfx12 with the numbers above. It covers
every context above 8K, and the format is not equal to 8-bit in quality at long context (+0.00530 KL,
t +3.44, top-1 −0.18 points at 131.060 positions) — the memory it frees is worth more there, but the
trade is named above instead of being left open. The format is
fixed (4.5 bit per cell, one fitted step per 32 values) — the ablations behind that, and the group-size
trade (64 values would save 0.25 bit per value and cost 0.92 dB), are in
`bench/results/2026-11-14-q4u32-fastpath/README.md` and the two papers in `docs/paper/`. The research modes
behind explicit flags (`q4lm32`, `q4lm8*`, `q4u32z`) come along for the record's reproducibility and are not
the default.

No site

Links install, modelos, releases.