#4859 [Engine] a 3-bit Lloyd-Max K/V (--kv oct3) with an exact fp16 window, and the gfx1100 decode ports
open · @destroyor · 0 Kommentare · Auf GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Beschreibung
**The title must start with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other], for example "[Engine] fix decode stop on long prompts" or "[GUI] keep the model list after the tab switch".** ## Problem and result An **RX 7900 XTX (gfx1100)** is the one RDNA branch `src/core/arch_defaults.cpp` skips: the matrix-core prompt set is gated on gfx1151 (Strix Halo), gfx1103 (the 780M) and gfx12 (RDNA4), so a discrete RDNA3 card runs the default path for everything. Two things fall out of that, both measured on one box: - Before: a 33,072-token prompt prefills at **1,880 tok/s** and the 16-prompt decode median is **90.00 / 90.65 tok/s**. - After, on the branch rebuilt on current `main` (`c62d5e3a`, `md5 e78294f3`) and deployed: **3,464.5 / 3,517 tok/s** prefill (+84%), 96K at **3,568** (+99%), decode median **109.5 tok/s** with **23.23 ms a forward** (from 24.91 / 25.55). That reproduces the numbers this branch measured before the rebase (3,467 / 3,517 prefill, 111.0 / 112.6 decode at 23.03 / 23.40): the rebuild on 170 newer commits cost nothing. One thing the re-test did catch is that the numbers depend on the **unit's full environment** - a launch that exported only part of it (no `GDN_*` group, `PF_F16`, `PF_HCDOWN`, `STAGER_RING`) read 3,084 / 3,165 prefill and 99.9 decode tok/s on the same binary. | | before (0.1.49.2 production) | after (rebased build, deployed) | |---|---|---| | prefill 33,072 tokens | 1,880 tok/s | **3,464.5 / 3,517** | | prefill 96K | 1,794 tok/s | **3,568** | | decode, 16-prompt median | 90.00 / 90.65 tok/s | **109.5** | | decode, `ms per forward` | 24.91 / 25.55 | **23.23** | | 128K prompt | - | 3,497 / 3,514 tok/s, 25.05 ms a forward | | 256K prompt | - | 3,240 / 3,260 tok/s, 25.96 ms a forward, 94.95% of KV reads served from VRAM | Long-context stability at that setting, same day: **82 of 82 requests over 192K-202K contexts** (single conversations of 6, 8 and 20 turns, then 2 x 12-turn and 3 x 8-turn concurrent conversations), on the pre-rebase build. RSS plateaus at 61.5 GiB for one conversation and 65 GiB for three under a 72 GiB cgroup, with no watchdog fire and no growth across 20 turns; an over-long request is refused with a clear 400 rather than truncated and the service stays up. A 200K prompt costs 60 s cold and 1.7-2.0 s a turn once the prefix is cached. ### Re-test on the deployed build (2 Nov 2026) Same card, same binary `md5 e78294f3` (engine 0.1.55), measured with the repository's own suite (`tools/strata_bench.py --runs 3 --long`, the community benchmark `docs/COMMUNITY_BENCHMARKS.md` points people at) rather than the 16-prompt harness: | | before (0.1.49.2) | PR (rebased, deployed) | re-test (official suite) | |---|---|---|---| | prefill 32K | - | 3,464.5 / 3,517 (33,072 tokens) | **3,456.1** (32,049 tokens) | | prefill 96K | 1,794 | 3,568 | - | | prefill 128K | - | 3,497 / 3,514 | ~3,235 (122,938 tokens, 38 s cold) | | prefill 250K | - | 3,240 / 3,260 | ~3,325 (242,749 tokens, 73 s cold) | | decode, 16-prompt harness | 90.00 / 90.65 | 109.5 at 23.23 ms a forward | - | | decode, official suite (chat / agent) | - | - | 95.1 / 102.8 tok/s | | decode, 256-token streaming, 24 rounds | - | - | 99.8 tok/s, 10.03 ms a token, 64.2% acceptance | Prefill 32K is within 0.2% of the PR number on the deployed build. The decode rows are listed separately because they differ by suite and by acceptance rate, which swings 59-76% with the prompt text; they are not one comparable column. Two notes from this pass. `--ple-io direct` on btrfs with `compress=zstd` stalls the batched prompt attention at a chunk boundary: 32 of 59 threads sat in state D on `__iomap_dio_rw`, no layer was served for 60 s, and the engine ended itself (#29); the log holds three such stalls from production traffic the day before. `--ple-io ram` ran six 128K/250K requests with no stall and the same 32K prefill speed, so the recipe below should read `ram` on a compressed filesystem. Needle recall on oct3 at that setting: 32K 3/3, 128K 3/3, 250K 3/3; five code words at 10/30/50/70/90% came back 5/5 in order, twice at 64K and twice at 250K (242.8K tokens, 74-75 s). ### Rebuild on v0.2.10 (3 Nov 2026) The branch is rebuilt on `ffa80d73` (v0.2.10): three commits, the same 53 files, the same box; the rebase touched six files (`CMakeLists.txt`, `src/core/verify.cpp`, `src/kernels/cuda/native_bf16.cu`, `src/prefill/prefill.cpp`, `src/program/generate.cpp`, `src/kernels/cuda/qsa_prompt_attn.cu`). The first rebuilt binary lost 32-44 % of the prefill; there were two causes, both fixed: 1. **Build config**: the fused MMQ expert path (`moe_fused.cu` / `moe_fused_iq.cu`) compiles only with `-DSTRATA_PREFILL_MMQ=ON`; a plain build left the fused gate/up path off. Now built with it (the service sets `STRATA_PREFILL_MMQ=1` to turn it on). 2. **The oct3 prompt attention was lost to the upstream refactor** (`7e4ac0c5`): v0.2.10 retemplated the wmma path into `prompt_attn_wmma_body<SPLIT,GATE,RPF,TR>` and gated its int8 entry on `pools.k_q4 != nullptr`, which the 3-bit pools never satisfy, and resolving the conflict took the upstream file wholesale. oct3 silently returned to the gather path: qsa attn 2,747 ms over the standard suite, 16K prefill 32-44 % down. The fix ports the oct3 matrix-core kernel as `prompt_attn_oct_kernel` with its own shared `vsb` (upstream's `SmemWT` is untouched), selected when `pools.oct3` is set - still opt-in (`STRATA_PA_OCT=1`, and on gfx11 it needs `STRATA_HIP_WMMA=1` beside it as before; `STRATA_PA_FAST=1` picks the wide-register variant). qsa attn is back to **486 ms**. Deployed and re-measured with the repository's own suite (`strata bench run --suite standard-v1`, corpus `e75eff8b`, engine 0.1.56 on the v0.2.10 base), oct3 + the exact window + resident 65536, next to the upstream DFlash2 drafter (DFlash2 itself is upstream's, not in this PR; four rounds of MTP / DFlash2 / no-drafter A/B before deployment put DFlash2 +4-10 % on decode): | | 0.1.55 deployed | this build (standard-v1, median of 3) | |---|---|---| | prefill 16K | 3,150+ tok/s | **3,171.2** (3,162.0-3,177.9) | | prefill 4K | - | **1,557.4** | | decode short / 4K / 16K | 99.4 / 89.9 / 91.0 | **108.2 / 96.2 / 88.7** (+8.9 / +7.0 / -2.5 %; the 16K dip is inside the run spread, most of the lift is DFlash2) | | sanity | 12/12 | **12 of 12** (instruction 4/4, retrieval 4/4, tool calls 4/4) | Quality is unchanged at the long end: 64K needle 5/5 and 250K needle 5/5 (242,839 tokens, prefill 3,380 tok/s, decode 97.8). The production service runs this build with `--kv oct3 --kv-resident 65536` and `STRATA_PA_OCT=1 STRATA_PA_FAST=1 STRATA_HIP_WMMA=1 STRATA_PREFILL_MMQ=1`. ## Related discussion None - no prior issue and no scope agreement. We read the rule and deliberately kept every part **opt-in** (one `--kv` name, one env var per port) and left `arch_defaults.cpp` alone, so the default path and every default are untouched; that is why this went straight to a PR. If a maintainer would rather have an issue first, or wants this split (oct3 is one problem, the gfx1100 decode ports are another), say so and we will split it - see Limits. ## Approach **oct3** (`--kv oct3`, opt-in) stores K and V as 3-bit codes on a Lloyd-Max grid over a 256-value block, behind the same Walsh-Hadamard rotation the repo's own rotated Q4_0 KV already ships, so `<Hq,Hk> = <q,k>` and the scores are untouched. 14 B per 32 values = **448 B a cell: 55% of k8v4, 78% of q4_0**. The cells that carry the attention mass stay in fp16 in a per-layer ring of `4 sinks + 1024 tail` rows, each row carrying a 4-byte position tag, so a row a newer position has reused is a miss and never a wrong value. The window costs 24.1 MiB over 12 layers - 3.6% of what the same cells cost in an fp16 cache - and is not part of a snapshot, so every restore invalidates the tags. Why it is needed: 35-80% of the attention mass lands on a handful of recent cells, and their quantization error does not average out over a sequence. Quality against fp16 K/V on the same engine (`tools/quant_quality.py`): | | top-1 | mean KL | |---|---|---| | oct3, no window | 91.63% | 0.05261 | | oct3, 4 + 256 | 94.03% | 0.03704 | | **oct3, 4 + 1024 (default)** | **95.27%** | **0.01851** | | k8v4 (int8 K + rotated Q4_0 V) | 93.31% | 0.03961 | At 256 tail cells oct3 already beats k8v4 at **half the bytes**. oct3 had no matrix-core prompt attention (the call refused it), so every oct3 prefill ran the chunk path: `STRATA_PA_OCT=1` adds the arm, **2,280 / 2,304 -> 3,443 / 3,513 tok/s (32K / 96K), +51%**, within 1-2% of `--kv int8` at 2.4x fewer bytes per cell, decode per forward unchanged. On gfx11 that arm needs `STRATA_HIP_WMMA=1` beside it: after the rebase the matrix-core prompt attention defaults **off** on gfx11 (`hip_wmma_mode_env()` returns -1 and only gfx12 keeps the old default), and without it oct3 silently stays on the gather path (33,072 tokens at 2,995 against 3,464.5 tok/s). **The decode ports** (each its own env var, all off by default) close the gfx1100 gaps the default sets leave: a multi-column Q6_K x q8_1 GEMV (`STRATA_RDNA_Q6K`, **+3.5%** decode), a two-launch hyper-connection read (`STRATA_GR_RDNA`, **+4%**), three window fusions (`STRATA_COMBINE_CPU` / `STRATA_WAIT_FUSE` / `STRATA_ROUTE_PUBLISH`, **+2%** once the window had shrunk), `--pcie-frac 0` (**1.4 ms a window**: the PCIe share's four expert nodes are empty at that ratio), the draft cumulative-probability floor for the MTP window (`STRATA_SPEC_CUM=0.4`, acceptance 60% -> 66% at the same cost a forward), and the q8_0 GEMV row policy. The 16-prompt median goes 95.60 -> 103.3 -> 112.6 on the pre-rebase build, and **109.5 tok/s at 23.23 ms a forward** on the rebased build - the rebuild moved nothing. Two measurements we are reporting rather than acting on, because changing a default needs your call: `STRATA_SELECT_WMMA` works on gfx1100 (the kernel already accepts gfx11 through `gfx_arch_is_gfx11_wmma()`) and is worth **+16% prefill at 96K** while selecting exactly what the warp scorer does (29,995 positions, top-1 100.00%, KL 0.00000) at **no decode cost** (23.54 ms a forward with it against 24.09 without, full env, one session per arm), so it stays opt-in and the deployment keeps it on; and the prompt switch group that gfx1151/gfx12 get by default is worth **+22% at 32K and +29% at 96K** here (1,864 -> 2,280 and 1,787 -> 2,304), with the batched prompt path keeping top-1 on all 20 prompts (mean KL 0.0170). Three **config-only** changes completed the picture, listed so the numbers above are reproducible: `--prefill auto:32768 -> 34816` (a 33,065-token prompt crossed the chunk rule and paid a **second** full pass over the experts: 3,131 / 3,211 -> 3,467 / 3,517 on the pre-rebase build, `wait copy` 461 -> 18 ms), `--expert-cache auto` with `--vram-reserve-mib 512`, and `--kv-resident 131072` (the deployment currently runs 65536, which is what a 33K-96K prompt needs). **The deployment recipe** (server config `args` plus the env the unit exports), for reproduce-ability: `--kv oct3 --kv-resident 65536 --prefill 34816 --expert-cache auto --spec 4 --pcie-frac 0 --ple-io direct` and `--vram-reserve-mib 512` in **both** `args` and `user_args` (`args` is overwritten from `user_args` at start, which cost us a day), with `STRATA_PA_OCT=1`, `STRATA_PA_FAST=1`, `STRATA_HIP_WMMA=1`, `STRATA_SELECT_WMMA=1`, the six gfx1100 ports (`STRATA_RDNA_Q6K`, `STRATA_GR_RDNA`, `STRATA_COMBINE_CPU`, `STRATA_WAIT_FUSE`, `STRATA_ROUTE_PUBLISH`, `STRATA_SPEC_CUM=0.4`) and the `GDN_CHUNKED` / `GDN_HEAD` / `GDN_CONVL2` / `GDN_NOY` / `GDN_PP=2`, `PF_F16=2`, `PF_HCDOWN`, `STAGER_RING=64` group. Leave the last group out and the same binary reads ~12% lower prefill (3,084 / 3,165); the environment, not the rebuild, is what those numbers depend on. **Sources we drew on** (nothing here is unattributed): vLLM's Octave KV for the structure and for keeping the projection-preserving scale instead of the least-squares one; Max (
Mehr auf der Site
Links zu Install, Modellen, Releases.