#2693 Feature request: KV cache quantization (`--kv int8` / `--kv q4_0`) for the Qwen3.6-35B-A3B (strata-q36) path

本文

The q36 path keeps its KV cache in fp32 (`src/qwen35`; 45,056 B per context cell). On a 12 GB card that is the wall for long contexts: at `--max-context 131072` the KV alone takes ~5.9 GB, so **no routed expert stays in VRAM at all** (engine log: `0 of 256 per layer in VRAM`). The Flash-Next path already offers `--kv int8` / `--kv q4_0` / `k8v4`; on the q36 path, setting `--kv int8` changes nothing (same KV size, same placement).

Setup: RTX 3060 12 GB, Ryzen 7 5800XT, 64 GB RAM, Windows 10, Strata 0.1.44.1 (ready-made CUDA 13 engine), `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` UD-Q4_K_XL (setup's pinned file, SHA-256 checked), config as setup writes it (`--native <gguf> --max-context N --prefill auto --mtp 2`).

Measured (greedy, 120-token answers, warmup + 2 runs each, after a fresh start):

| context | decode | experts in VRAM | note |
| --- | --- | --- | --- |
| 65,536 | ~46 tok/s (45.5–47.1) | 18 of 256 per layer (1.32 GiB) | rest on the 8-thread CPU pool |
| 131,072 | ~34 tok/s (33.7–34.8) | **0 of 256 — nothing fits** | fp32 KV ~5.9 GB |

- MTP (`--mtp 2`) is worth +40–44% here (46 vs 32 tok/s at 64K; 34 vs 24 at 128K) — it stays on.
- In long filled conversations decode drops further (~30 tok/s; GPU ~50% utilised, memory controller ~36% — the bottleneck is the CPU/offload side plus the KV traffic).
- On the same PC, llama.cpp (via LM Studio, Q4_K_M, 128K context, quantized KV) holds ~43 tok/s at 128K — the KV format is the visible difference.

Why this fits the existing code: Qwen3.6's `head_dim` is 256 — the exact size of the FWHT-256 rotation used by `--kv q4_0` (PR #21). With int8 (1,056 B per cell) or q4_0 (576 B per cell) the 128K KV shrinks from ~5.9 GB to ~1.4 GB / ~0.76 GB: ~4.5–5.1 GB back for experts at 128K and 3–6x less KV traffic per decoded token (related: #2464, where decode falls with context on a V100).

Happy to run A/Bs (int8 vs q4_0 vs fp16, several contexts) on the 3060 if that helps.

---