Issues / #2594

#2594 KV streaming on a layer split reports `0 block reads` (0.1.43 `7695859a`, IQ2_XS, RTX 3090 + RTX 3060), while the same config on one card reports 215k-252k

open · @TurtleFlyRU · 0 comentários · No GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descrição

## What happens

On a layer split the engine's per-request counter reads `KV streaming: 100.00% of 0 block reads hit VRAM, 0.0 MiB
read from RAM` for every request, at every context size and with every KV format. The start line says three
quarters of each QSA layer's cells are not in VRAM, so the RAM copy should be read. The same machine, build, model
and `--kv-resident 32768` on the 3090 **alone** reports real reads for the same residency line.

Start line, split:

```text
strata generate: KV streaming: 32768 of 131072 cells per QSA layer in VRAM, the K/V in 0.13 GiB of pinned RAM
```

Start line, one RTX 3090, same `--kv-resident 32768`:

```text
strata generate: KV streaming: 32768 of 131072 cells per QSA layer in VRAM, the K/V in 1.55 GiB of pinned RAM
```

Per request, split (131072 context, a 4019-token prompt):

```text
strata serve: KV streaming: 100.00% of 0 block reads hit VRAM, 0.0 MiB read from RAM
```

Per request, one RTX 3090 (same context, the same 4K prompt in the same suite):

```text
strata serve: KV streaming: 89.58% of 246456 block reads hit VRAM, 103.5 MiB read from RAM
strata serve: KV streaming: 85.45% of 215652 block reads hit VRAM, 126.4 MiB read from RAM
strata serve: KV streaming: 87.18% of 252636 block reads hit VRAM, 130.4 MiB read from RAM
```

The split also reports 0 reads at `--max-context 262144` with prompts of up to 255,433 tokens, and with
`--kv k8v4` and `--kv q4_0`, so it is not one format, one context size or one prompt.

## Why I think something is off

With 32768 of 131072 cells per QSA layer in VRAM, 98304 cells per layer are not, and a 64K-256K prompt has to
attend to them. If the host copy holds them, those blocks have to be read from RAM. The split reports 0 such reads
in every run we made; the single card reports 200k-250k per request for the same residency. Either the split does
not stream K/V blocks at all, or the counter is not wired for the split path - I cannot tell which from the
outside.

A second thing I cannot explain and would like to understand: `1.55 / 0.13 = 12`. If the single card's number is
the whole model's host copy (12 QSA layers) and the split's number is one QSA layer's share, the two are the same
1.55 GiB and only the counter differs. If both numbers are the same quantity, then on a split the host copy is 12x
smaller, which would explain 0 reads but would mean the split keeps the K/V somewhere else. Which is it?

## Configuration

Split (both cards), port 8092:

```text
strata --serve --pack Strata-data/packs/iq2_xs
  --native .../Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf
  --ple-gguf .../Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf
  --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto
  --spec 4 --spec-min-p 0.5 --mtp Strata-data/mtp/rt
  --max-context 131072 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700
```

`"gpu": "0,1"`; the engine reorders to `card order 1,0`, so CUDA0 is the RTX 3060 (layers 0-2, 1536 slots,
2.06 GiB) and CUDA1 the RTX 3090 (layers 3-47, 12927 slots, 17.32 GiB). `--layer-split auto` chose K=3.

Single card: the same file with `"gpu": 1` (the 3090), everything else identical.

## How to reproduce

```bash
python tools/strata_bench.py --config <the split config> --port 8092 --api-key <key> --long --huge
python tools/strata_bench.py --config <the 3090-only config> --port 8092 --api-key <key> --long --huge
```

The counters are the `strata serve: KV streaming: ...` lines in the engine log, one per request, and the
`strata generate: KV streaming: ...` line at start. `tools/ctx_sizes.py --sizes 170000,262144` shows the same
thing at 262144.

## System

- Strata 0.1.43, commit `7695859a`, source build (engine 0.1.43)
- 2 GPUs: RTX 3090 24 GB (PCIe 4.0 x16, 250 W) + RTX 3060 12 GB (PCIe 3.0 **x4**, 130 W); driver 580.178.04, CUDA 13.0
- AMD Ryzen 7 7700X, 122.6 GiB RAM, Linux 7.0.0-31
- Model Qwen3.8-Flash-Next GSQ-RCO IQ2_XS (native pack), 24576 experts, shipped expert profile

## What I ruled out

- Not the KV format: int8, k8v4 and q4_0 all report 0 reads on the split.
- Not the context size: 131072 and 262144 both report 0 reads; the prompts were up to 255,433 tokens.
- Not the cache: the split's expert-cache hit rate over the same suite is 91-99%, the same as the single card's.
- Not `--kv-resident` being ignored: the start line does report `32768 of 131072 cells per QSA layer in VRAM` in
  both cases, and raising it to 65536 costs 294 slots on the 3090, so the option is read.

## What I did not test

- Another split (the same pair in the other order, or a pair of equal cards) - I can send those logs if they help.
- Windows, WSL and AMD.
- `STRATA_SPLIT_TIMING=1` on a long request, which is probably the next step - I can run it on request.
- Whether the 3.4 GB/s link of the 3060 (Gen3 x4) is involved.

## Related

- #2415 (kv_stream page-pool eviction) - same subsystem.
- #2007 (256K + KV streaming on a 2x T10 layer split): their prefill moves between engine versions with
  `--kv-resident 32768` on a split, so I would expect the streaming path to be active there; they do not quote
  the block-read counter.
- #2418 (x16+x4 link-gap rig class) - our rig has the same 4x-ish link gap, 26.8 vs 3.4 GB/s probed.
- #2450 (open PR: no KV streaming when one card holds every expert and the whole K/V) - that is the opposite
  case, a card that should not stream; this report is about the counter on a split.

I have per-run JSON, engine logs and the full configs for everything above and can attach them.

No site

Links install, modelos, releases.