Issues / #2960
#2960 Windows HIP, 0.1.44.1, one RX 7900 XTX: about 1 start in 5 the KV streaming does no block reads, and every prompt past --kv-resident gets a wrong answer
open · @jerem91150 · 0 Kommentare · Auf GitHub
BenchmarksServer & APIAMD / HIPModels & quantsWindows
Beschreibung
Native Windows 11, one RX 7900 XTX (gfx1100), 64 GB RAM, Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, 0.1.44.1 HIP build, `--max-context 131072 --kv int8 --kv-resident 32768`, MTP draft (`--spec 4`), `--expert-cache auto`, `--vram-reserve-mib 1536`. **What happens.** Some engine starts answer every long prompt wrong. The prompt is read in full (`0 reused`), the speed looks normal, but the answer has nothing to do with it. The test is a fresh ~66K-token prompt (Python source files) ending with "which port does the router listen on (the PORT variable)? Answer with the number only", then a fresh ~113K one with the same question. The answer is in the text (18802). Good starts answer `18802` both times. Bad starts answered things like: ``` Based on the image provided, here is a description of the scene: ... ### 优化后的提示词 ... The following analysis provides a comprehensive and structured response to the user's query ... ``` (no image was sent), and the follow-up in the same conversation went "the code visible in the image is incomplete". **It is decided at start, and the log shows which start is bad.** In the bad starts the KV streaming counter reads zero from the first request on: ``` strata generate: KV streaming: 32768 of 131072 cells per QSA layer in VRAM, the K/V in 1.55 GiB of pinned RAM strata serve: prompt 20 tokens = 0 reused + 20 read in 320 ms ... strata serve: KV streaming: 100.00% of 0 block reads hit VRAM, 0.0 MiB read from RAM strata serve: prompt 65960 tokens = 0 reused + 65960 read in 58660 ms (1124.4 tok/s) ... strata serve: KV streaming: 100.00% of 0 block reads hit VRAM, 0.0 MiB read from RAM ``` while a good start, same config, same prompts: ``` strata serve: KV streaming: 96.30% of 3240 block reads hit VRAM, 0.5 MiB read from RAM strata serve: prompt 65960 tokens = 0 reused + 65960 read in 58132 ms (1134.7 tok/s) ... strata serve: KV streaming: 74.40% of 92424 block reads hit VRAM, 95.3 MiB read from RAM ``` Startup lines are the same in both (same expert cache: `17.89 GiB free ... -> 7472 slots`, same residency line). **Counts, 16 fresh starts today, alternated:** | config | starts | starts with `0 block reads` | starts with wrong answers | |---|---|---|---| | `--kv-resident 32768` (K/V past 32K in RAM; 2 of these with `STRATA_HIP_WMMA=1`, both good) | 10 | 2 | 2 (the same two) | | `--kv-resident 131072` (all K/V in VRAM) | 6 | - (no streaming) | 0 | So for us the wrong answers and the zero counter go together 10 out of 10 times. Short prompts (we never saw a wrong answer under ~27K) look fine even in a bad start, so it only shows once a conversation goes past the resident part. This looks related to #2594 (`0 block reads` on a layer split), except here it is one card, it only happens on some starts, and the answers are wrong, so it is not just the counter. Workaround for now: `--kv-resident 131072`. On this card it costs ~2% on fresh long prompts (66K: 59.0 against 58.1 s, 113K: 103.7 against 101.8 s), decode at 113K the same. I can send full engine logs of a good and a bad start, or run a build with extra logging.
Mehr auf der Site
Links zu Install, Modellen, Releases.