#3360 [Benchmark] RTX A5500 24 GB: KV format, parallel slots and --vram-reserve-mib (engine 0.1.45)

Descrição

Measured on 2026-10-20 by @wishmeng. Six configurations, each on its own freshly started engine, measured with `tools/strata_bench.py --long` (three runs each). Main limitation: a speed report on one card, with a locally prepared pack and a locally learned expert profile - no recall or quality test, no `--clients N` concurrency test, no GPU clock or PCIe capture.

## Hardware and software

- GPU: NVIDIA RTX A5500 (24564 MiB, sm_86); CPU: Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz (10 cores / 20 threads, no AVX-512); RAM: 127.7 GiB; storage: NVMe (about 79 GB of model files). PCIe link speed and width: not measured. GPU power limit: default.
- OS: Windows 11 (10.0.26200); driver 595.95; CUDA 13.2; Python 3.14.8
- Strata 0.1.45, engine 0.1.45 (official Windows release binary, prebuilt for sm_75/86/89/120)
- Background workloads during a run: none. Each variant ran on a freshly started engine after the model had loaded and 240 s of idle time, with no other client on the server.

## Model and configuration

- Model: Qwen3.8-Flash-Next, IQ3_XXS, native-experts pack `iq3_xxs`, split GGUF `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf` + `...-00002-of-00002.gguf`; the PLE table is read directly from the second file. Locally prepared pack: name and layout given, files not published.
- Expert profile: locally learned `expert-profile.bin`, 0.2 MB, sha256 `8f59b4aa8873209dff11c11e37bcda9529a1335b724a1afeea37bf6388975baf` (24576 ranked pairs, built for 24576 slots). The file itself is not published; every number here depends on it.
- Common to all six: `--max-context 262144`, `--expert-cache auto`, `--prefill auto`, `--spec 4 --spec-min-p 0.5`, `--mtp <MTP_DIR>`, `--vision` (CPU vision encoder), server on 127.0.0.1:8080, thinking off for the suite.

| Variant | `--kv` | `--kv-resident` | `--vram-reserve-mib` | parallel slots |
| --- | --- | ---: | ---: | ---: |
| A | `q4_0` | 32768 | 966 | 2 |
| B | `k8v4` | 32768 | 966 | 2 |
| C | `k8v4` | 16384 | 966 | 2 |
| D | `q4_0` | 32768 | 966 | 1 |
| E | `int8` | 32768 | 966 | 1 |
| F | `q4_0` | 32768 | 1536 | 2 |

```text
strata.exe --serve --pack <MODELS_DIR>/packs/iq3_xxs --native <MODELS_DIR>/models/IQ3_XXS/<model>-00001-of-00002.gguf
  --ple-gguf <MODELS_DIR>/models/IQ3_XXS/<model>-00002-of-00002.gguf
  --expert-profile <STRATA_DIR>/data/expert-profile.bin --expert-cache auto --prefill auto
  --spec 4 --spec-min-p 0.5 --mtp <MTP_DIR> --max-context 262144
  --kv <q4_0|int8|k8v4> --kv-resident <32768|16384> --vision --vram-reserve-mib <966|1536>
```

## Method

- One engine process per variant: the running engine was stopped, the variant config started a new server on `127.0.0.1:8080`, `/props` was polled until the model reported ready, then 240 s of idle time with no requests.
- Suite: `python tools/strata_bench.py --long --config <variant config> --compare <variant A folder>` - three short chats capped at 100 new tokens, fresh word-list prompts of about 4,096 / 16,384 / 32,768 tokens (each carries a unique tag, so nothing is reused), and a six-turn agent session with four tools.
- Throughput is the engine's own `prompt_per_second` / `predicted_per_second`; TTFT is measured by the client. Medians with min-max ranges over three runs.
- Memory figures are the engine's startup snapshot (`N MiB of VRAM free with everything loaded`, `expert cache N slots`) and its per-request low-memory warnings - not peak-during-inference values.
- State between runs: fresh process each time, so no conversation prefix carried over; the expert profile and the profile-driven cache pre-fill are the same file in every variant.

## Results

| Variant | chat decode tok/s | chat TTFT s | 4K prompt tok/s | 16K prompt tok/s | 32K prompt tok/s | 32K TTFT s |
| --- | --- | --- | --- | --- | --- | --- |
| A | 69.30 (51.3–76.1, n=9) | 0.48 (0.4–0.6, n=9) | 1389.30 (1303.4–1393.3, n=3) | 2266.80 (2195.8–2274.4, n=3) | 2231.00 (2217.3–2238.9, n=3) | 14.43 (14.4–14.5, n=3) |
| B | 62.20 (47.9–72.7, n=9) | 0.51 (0.4–0.7, n=9) | 1365.20 (1274.1–1366.0, n=3) | 2128.90 (2063.9–2139.0, n=3) | 2123.00 (2107.5–2128.8, n=3) | 15.17 (15.1–15.3, n=3) |
| C | 64.20 (50.0–72.8, n=9) | 0.50 (0.4–0.6, n=9) | 1374.40 (1292.7–1376.1, n=3) | 2134.20 (2056.8–2151.1, n=3) | 2117.70 (2111.6–2120.3, n=3) | 15.23 (15.2–15.2, n=3) |
| D | 74.20 (64.0–92.6, n=9) | 0.40 (0.3–0.4, n=9) | 1456.80 (1358.8–1459.8, n=3) | 2236.80 (2212.1–2260.4, n=3) | 2220.70 (2216.4–2226.5, n=3) | 14.48 (14.4–14.5, n=3) |
| E | 72.20 (59.2–93.6, n=9) | 0.40 (0.3–0.5, n=9) | 1448.30 (1351.7–1450.7, n=3) | 2453.20 (2381.2–2459.8, n=3) | 2402.50 (2401.5–2411.2, n=3) | 13.39 (13.4–13.4, n=3) |
| F | 61.60 (52.5–75.3, n=9) | 0.52 (0.4–0.6, n=9) | 1364.20 (1279.0–1365.7, n=3) | 2231.90 (2155.2–2247.1, n=3) | 2219.70 (2202.6–2226.5, n=3) | 14.49 (14.4–14.6, n=3) |

| Variant | agent turn decode tok/s | agent turn TTFT s | cache hit % | re-read tokens / session | last-turn prompt tokens |
| --- | --- | --- | --- | ---: | ---: |
| A | 75.00 (50.6–96.0, n=15) | 3.08 (2.8–3.5, n=15) | 68.67 (68.7–68.8, n=3) | 74.00 (2.0–74.0, n=3) | 14594 |
| B | 77.50 (45.9–97.8, n=15) | 3.17 (2.9–3.6, n=15) | 68.59 (68.5–68.9, n=3) | 74.00 (4.0–144.0, n=3) | 14571 |
| C | 79.10 (57.5–97.3, n=15) | 3.17 (2.9–3.6, n=15) | 68.69 (68.6–68.7, n=3) | 73.00 (73.0–74.0, n=3) | 14580 |
| D | 73.40 (35.1–94.2, n=15) | 2.94 (2.7–3.4, n=15) | 68.73 (68.6–68.8, n=3) | 4.00 (3.0–74.0, n=3) | 14540 |
| E | 86.00 (63.5–110.0, n=15) | 2.91 (2.7–3.4, n=15) | 68.69 (68.6–68.9, n=3) | 73.00 (4.0–74.0, n=3) | 14598 |
| F | 74.30 (58.3–88.6, n=15) | 3.14 (2.9–3.6, n=15) | 68.87 (68.8–68.9, n=3) | 3.00 (2.0–3.0, n=3) | 14614 |

Token counts are the same in every variant: 4019 / 16058 / 32049 prompt tokens with 0 reused, 100 generated tokens per chat request, about 13718 new tokens per agent session.

### What the engine reported about memory

| Variant | expert cache slots | cache VRAM | auto sizing | KV resident cells / QSA layer | KV host copy (pinned) | prompt path borrows | VRAM free at start | low-memory warnings | lowest warning value | decode cache hit % | KV streaming VRAM hit % |
| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| A | 9024 | 14.62 GiB | 15.74 GiB free, 966 MiB reserved (+184 MiB for the draft head) -> 6737 slots | 32768 | 1.69 GiB | 2546 (4.14 GiB) | 518 MiB | 2 | 378 MiB | 91.0 | 98.4 |
| B | 8855 | 14.34 GiB | 15.46 GiB free, 966 MiB reserved (+184 MiB for the draft head) -> 6609 slots | 32768 | 2.39 GiB | 2586 (4.19 GiB) | 520 MiB | 2 | 349 MiB | 90.6 | 98.4 |
| C | 8980 | 14.54 GiB | 15.80 GiB free, 966 MiB reserved (+184 MiB for the draft head) -> 6766 slots | 20480 | 2.39 GiB | 2583 (4.20 GiB) | 611 MiB | 2 | 468 MiB | 90.6 | 98.4 |
| D | 9971 | 16.16 GiB | 17.29 GiB free, 966 MiB reserved (+184 MiB for the draft head) -> 7450 slots | 32768 | 1.69 GiB | 2530 (4.14 GiB) | 503 MiB | 2 | 372 MiB | 95.2 | 98.3 |
| E | 9853 | 15.97 GiB | 17.10 GiB free, 966 MiB reserved (+184 MiB for the draft head) -> 7363 slots | 32768 | 3.09 GiB | 2600 (4.25 GiB) | 509 MiB | 2 | 364 MiB | 94.6 | 98.4 |
| F | 8680 | 14.06 GiB | 15.74 GiB free, 1536 MiB reserved (+184 MiB for the draft head) -> 6480 slots | 32768 | 1.69 GiB | 2554 (4.14 GiB) | 1090 MiB | 2 | 887 MiB | 90.2 | 98.5 |

Every variant printed the engine's low-memory warning at least once (`WARNING: only N MiB of VRAM free at the start of this request (below 1024 MiB) ... (#2065)`). None of these runs degraded to a few tok/s, so on this card the warning marked a risk rather than an observed stall.

## What these numbers say

- **`--vram-reserve-mib 1536` is slower than 966 here.** A vs F: chat decode 69.30 -> 61.60 (-11%), expert cache 9024 -> 8680 slots, decode cache hit rate 91.0% -> 90.2%. It buys headroom instead of speed: 518 -> 1090 MiB free at start, lowest warning value 378 -> 887 MiB.
- **One slot is faster than two.** D vs A: chat decode 69.30 -> 74.20 (+7%), chat TTFT 0.48 -> 0.40 s, agent turn TTFT 3.08 -> 2.94 s. The slot sessions go back to the expert cache: 9024 -> 9971 slots, decode cache hit rate 91.0% -> 95.2%.
- **`--kv int8` beats `--kv q4_0` on long prompts and multi-turn work at one slot, not on short chat.** E vs D: 16K prompt 2237 -> 2453 (+9.7%), 32K prompt 2221 -> 2403 (+8.2%), 32K TTFT 14.48 -> 13.39 s, agent turn decode 73.4 -> 86.0 (+17%), while chat decode is level to slightly worse (74.2 -> 72.2). The cost: pinned KV host copy 1.69 -> 3.09 GiB, and 9971 -> 9853 slots.
- **`--kv k8v4` did not win at two slots on this card.** B vs A: chat decode 69.30 -> 62.20 (-10%), 16K prompt 2267 -> 2129 (-6%); it only helped the agent session (75.0 -> 77.5). Halving `--kv-resident` (C) recovered the memory - 8980 slots, 611 MiB free at start, 468 MiB lowest warning - but not the speed (64.20 chat decode).

## Correctness and limitations

- No quality measurements: no needle/recall test, no coding task, no perplexity. The published note that `q4_0` KV costs precision on long documents (perplexity +8-12%) was not checked here.
- No concurrency measurement: `--clients N` was not run, so the two-slot variants are compared only on sequential requests; the single-slot variants would serialise two simultaneous clients.
- One suite session per variant (three repetitions inside it), no repeat sessions on other days, no order-swapped pairs.
- Memory numbers are engine startup snapshots and warning lines, not peak VRAM during inference; no `nvidia-smi` sampling, so paging was neither measured nor ruled out.
- The expert profile and the locally prepared pack are part of every number; a machine with a different profile can rank these variants differently.
- Not tested: `--max-context` below 262,144, other `--prefill` sizes, `STRATA_DF_QB_Q4=0`, `STRATA_QSA_SPLIT_ATTN=1`, the 0.2.0-beta engine, and whether a reserve between 966 and 1536 beats both.

Per-run data: the six `bench/results/` folders named in `my/kvtest/runs.csv` (each `results.json` holds every request and the machine facts), plus one engine log per variant.

### Reading the two slot numbers

`expert cache auto: ... -> N slots` is what the free VRAM alone would allow; with a profile the engine then fills more slots than that (`expert cache 9024 slots, 14.62 GiB` in the next line). The table gives both, and the speed rows should be read against the final slot count.