#6175 [Engine] the 4-bit KV cache (q4u32) as the default on RDNA4 — scope agreement
open · @monscoro · 1 comentários · No GitHub
BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsWindows
Descrição
## The user problem Above 8K context the engine keeps the KV cache at 8 bit (8.25 bit per value with the per-row scale) — what every published number was measured with. On an RDNA4 card that memory is worth more in the expert cache than in the KV cache: the cache is the model's memory of the conversation, and every byte it does not need holds another resident expert instead. The 4-bit mode Strata has (`q4_0`, #21) does not use the room it has: it derives a block's step from the block maximum, and after the Hadamard rotation the values are roughly Gaussian, so the maximum is a poor scale. The format that fits the rotated values is the same grid with a **fitted** step. ## Proposed behavior Without `--kv` the engine picks **`q4u32`** where its prompt path is the measured one: gfx12 with the gfx12 prompt defaults. Everywhere else the default stays 16-bit, and a format the person named is always written, so a start outside that one case is byte-identical to the current release. Measured on one card (AMD Radeon RX 9060 XT 16 GB, gfx1200, Windows, ROCm 7.2, 32K): - prompt rate level with `int8` (664.5 against 669.0 tok/s; the paired A/B of the coupled switch: −1.8 % mean, −2.1 % median) - 54.5 % of `int8`'s bytes per cell (144 against 264) — the expert cache takes the freed memory as **+18.7 % more resident expert slots** - without the matrix-core prompt path the same format is 2.5× slower (265 against 655 tok/s), so the path is the precondition for the default and the default is limited to the parts that have it ## Tradeoffs - The default changes on gfx12 only. A part that would run the slower route (gfx11, CUDA, an upstream engine) does not get the format by default; `STRATA_HIP_WMMA=0` keeps 16-bit everywhere. - The format is fixed: 4.5 bit per cell, one fitted step per 32 values, no codebook, no zero point, no calibration. The group size 64 would save another 0.25 bit per value and costs 0.92 dB — not taken. - The engine names itself with a variant suffix (`0.1.61-q4u32`), and an update never switches the installed variant, so a person who has the build keeps it. - The measurements are one card family and one card. The RDNA3 and APU arms (gfx11, Strix Halo / 780M) are pre-registered and not measured — no such card here. #5535 is the same question on RDNA3. ## Related #21 (the `q4_0` mode and the Hadamard rotation), #329 (the gfx12 WMMA prompt kernel), #5535 (RDNA3, open). A PR with the change and the measurements follows once the scope is agreed.
No site
Links install, modelos, releases.