#2450 setup: no KV streaming on one card that holds every expert and the whole KV cache (#2261)

本文

Fixes #2261.

From 64K up setup writes `--kv-resident 32768` whenever the RAM holds the K/V copy. On a card that already holds every expert and the whole K/V in VRAM, the VRAM this frees has nothing left to hold, and the cells past 32K are read from RAM.

With `--kv-streaming auto`, setup now keeps the K/V in VRAM on one card (no layer split, not a shared-memory GPU) when its VRAM covers the experts (`arena_gb`), the whole K/V of each session (1 + the `parallel` slots, from `--parallel` or the installed config) and 8.5 GB beside them. The 8.5 GB is what the engine used beside the experts and the K/V on an L40S (6.9 GB), its 700 MiB reserve, and about 1 GB for a desktop. Only the native packs' sizes: Unsloth's and the NVFP4 files keep the RAM test alone, since their dense side was not measured. `--kv-streaming on|off` work as before, and `kv_streaming_wanted` gives the same answer as Step 7. At 262K this turns streaming off for Q2_0 / IQ2_XS / IQ1_M on 48 GB cards and for every native size on 80 and 96 GB cards.

A/B on one L40S (46068 MiB, sm_89, Linux), engine 0.1.44.1 (296b59d), IQ2_XS, `--max-context 262144 --kv int8`: `--kv-resident 32768` (S) against no `--kv-resident` (R). Every expert is in VRAM in both arms (24576 of 24576; 5.4 vs 2.4 GiB free). One warm-up start, then S R / R S / S R on two Modal hosts. Engine timings from the replies, medians of 3:

| host | request | S | R |
|---|---|---:|---:|
| A | 48K-token prompt, read | 4327 tok/s | 4580 tok/s |
| A | 120K-token prompt, read | 4385 | 4738 |
| B | 48K / 120K prompt, read | 4616 / 4789 | 4616 / 4837 |
| B | 3 short chats, decode | 231.7 / 150.5 / 166.0 | 237.0 / 153.9 / 168.9 |
| B | decode after the 48K / 120K prompt | 140.4 / 134.8 | 147.0 / 141.1 |

On host B both arms of a round read the same prompts, so decode compares too (R was faster in 15 of 15 pairs). On host A the long prompts were new at each start, so only their read speed compares (R faster in 6 of 6). Host A's short chat and host B's 48K read were within 1% either way. The gains are smaller than beastrobin's on the RTX PRO 6000, in the same direction. R also pins 3.09 GiB less RAM.

Left as it is: IQ3_XXS on the L40S, where not every expert fits, still streams. There streaming gets 22991 expert slots against 21163, but the long prompts still read about 7% faster without it on host A; decode was too noisy to tell.

Tests: two new tests in `tools/test_setup_risk.py` (KvStreaming). The RTX PRO 6000 with IQ3_S and the L40S with IQ2_XS keep the K/V in VRAM; `--kv-streaming on`, IQ3_XXS on the L40S, `--parallel 2`, two L40S in a split, UD-Q4_K_XL and NVFP4 still stream. The first fails on main. All `tools/test_setup_*.py` on Linux (Python 3.12): the same 10 failures on main and on this branch (the UD-Q4_K_XL goldens, and two engine_hash tests that look for a release download), and the 2 new tests pass.

Not tested: Windows, AMD, the RTX PRO 6000 itself, `--parallel` or the GPU image encoder on a big card, `--kv k8v4` and `q4_0`.