#3678 [Engine] q36: quantized KV caches behind --kv (int8 / fp8 / rk4v4-e8 q4)

open · draft · @yang478 · 0 commentaires · Sur GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindows

Description

**Area:** [Engine]

## Problem and result

The q36 engine (src/qwen35) kept the KV cache in fp16 only. At head dim 256 with GQA 16:2 that is 2 KiB per token per KV head pair, so on a 12 GB card a 150K- or 180K-token window leaves no room for a useful expert cache — the long-context Ornith-1.5-35B-A3B runs were simply not possible.

With this PR the engine builds its KV cache in three quantized forms behind the existing `--kv` argument, opt-in per start, fp16 unchanged as the default and bit-identical to before:

- `--kv int8` — int8 codes with a f16 scale per 64 values: 528 B per cell and KV head pair at HD 256 (26% of fp16).
- `--kv fp8` — E4M3 codes with a f16 scale per cell and KV head: 516 B.
- `--kv q4` (`rk4v4-e8`) — K and V packed int4 over a normalised 64-point Hadamard rotation, with an E8 lattice projection on K (`STRATA_Q36_Q4_NOE8=1` drops the projection, the A/B switch): 544 B.

## Related discussion

none opened yet — if a scope discussion is preferred for this size, happy to move it to an issue. The measured evidence (150K/180K fresh prompts, int8 vs fp8, needle 10/10, MTP identity) is in the companion benchmark PR #3677; the Q4_K expert down formats that Ornith also needs are in a separate small PR.

## Approach

- `src/qwen35/q36_engine.cu`: `KvType {F16, Q8, FP8, Q4}`; the quantized caches ride the same `k_attn_prep` / `k_attn_part` shape, rewritten to read and write codes (`k_attn_prep_q8` / `k_attn_prep_fp8`, and the fused q4 path, which is written for G = H / HKV = 8 as in this family). The prompt's GEMMs read a compact f16 staging (`k_kv_f16`) written from the codes once per window. The GQA partial-attention kernel now walks the KV cache in chunks resident together so the query heads of one chunk share the K/V reads. `--kv` parsing, the startup `KV cache:` log line (type, bytes/token, scales) and the per-type staging estimate live here too.
- `src/qwen35/q36_kernels.cuh`: the quantization arithmetic — `kv_q8` (scale fp16(max|x|/127), codes rounded against the stored f16 scale), E4M3 encode/decode written out (largest value 448, `0x7F` never emitted) and the H64 Hadamard — one definition shared by the CUDA and HIP builds, no `cuda_fp8.h` (the sm_89+ hardware instruction is used where available behind the same inline).
- Opt-in everywhere: without `--kv` nothing changes. Two extra opt-in knobs came with the work and are included because the KV sizing motivated them: `STRATA_Q36_TRIM=1` pushes the process's cold pages to the standby list after load (working set 41 → 7.7 GB on the 64 GB machine; costs decode, off by default), and `STRATA_Q36_Q4_NOE8` above.
- The diff is 2 files, +1191/-66, based on main `011c9e6`.

## Validation

Hardware: NVIDIA GeForce RTX 4070 Ti 12,282 MiB (driver 617.14), 13th Gen Intel Core i7-13700K, 64 GB RAM, Windows 11 (10.0.26200); CUDA 12.8, sm_89, MSVC 2022 BuildTools + Ninja.

- **Passed:** these engine sources (the same two files plus local bench tooling, pre-rebase tree `266e787`) built cold and ran the full measured day behind benchmark PR #3677 — 150K and 180K fresh prompts under int8 and fp8, needle retrieval 10/10 including both 180,067-token rows, greedy MTP0/MTP1 answers byte-identical at 2K.
- **Passed:** `strata smoke` (2K fixture, needle) on the same tree.
- **Passed:** cold build of this exact branch (rebased onto `011c9e6` + this PR): `cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89 -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_RUNTIME_LIBRARY=Shared -DSTRATA_BUILD_TESTS=OFF` then `ninja strata-q36` — 158/158 steps, links, and the binary starts: `strata-q36 --version` prints `--kv fp16|int8|fp8`.
- **Not run:** the HIP and SYCL builds (no AMD/Intel GPU or toolchain here). The kernels header was written so CUDA and HIP share one definition, but that is a claim about the code, not a tested build. Keeping the PR as a draft for that; if a maintainer wants it, the HIP compile is the missing check I would need help with.

## Limits

`--kv q4` and `--kv fp8` change the numbers (see #3677: fp8 costs 11-16% decode against int8 at 150K/180K, prefill at parity); q4 rows are measured only on the pre-rework binary so far and are not in #3677's tables. The fused q4 attention path assumes G = 8 (the family's GQA ratio); other ratios fall back to the staged path. `STRATA_Q36_TRIM` trades decode for host working set and stays off by default.

Sur le site

Liens install, modèles, releases.