#2735 NVFP4 prompt modes on a 96 GB Blackwell card (every expert in VRAM): w4a4 reads fresh prompts 9-13% faster than w4a8

Descripción

**Strata version, GPU, OS:** 0.1.44.1 (`296b59d2`, release engine `strata-windows-x64.zip`, its CUDA 13.0 build) · RTX PRO 6000 Blackwell Workstation Edition 96 GB (sm_120, PCIe 5.0 x16), driver 616.92 · Windows 11 (26200) · i9-14900K, 192 GB DDR5-5200

[docs/NVFP4.md](https://github.com/Niko1221/Strata/blob/296b59d2f73c8f04a805df29c2fa97655adace78/docs/NVFP4.md) keeps `w4a8` the default "until a 96 GB+ RTX 50 PC measures the others". On this card the engine logs `every expert is in the GPU cache (24576 of 24576)` (63.29 GiB), so no prompt waits for RAM or the SSD.

**Setup.** `--family nvfp4 --model NVFP4` (the GPTQ pack, `Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata` @ `640a1db9`, every file at setup's pinned SHA-256) with setup's plan for this card: `--max-context 262144 --kv int8 --kv-resident 32768 --expert-cache auto --spec 4 --spec-min-p 0.5 --pool-workers 15`, `--parallel 4`, and the two settings setup recommends, `--prefill auto:32768 --conversation-cache-mib 8192`; text only. The `w4a4` arm is the same config with `"env": {"STRATA_PREFILL_NVFP4": "w4a4"}` (its log has no W4A8 fallback note). Reference: IQ3_S (ISTA GSQ-RCO) with the same settings. Each arm started cold on an idle GPU, then `python tools/strata_bench.py --long --clients 4` ran against it: 3 runs per test, medians with ranges, engine timings.

| | IQ3_S | NVFP4, `w4a8` (default) | NVFP4, `w4a4` | w4a4 vs w4a8 |
|---|---:|---:|---:|---:|
| 4K prompt tok/s | 5,272 (5,175-5,281) | 4,957 (3,804-4,965) | 5,603 (4,218-5,698) | +13.0% |
| 16K prompt tok/s | 7,345 (7,259-7,365) | 7,010 (5,951-7,024) | 7,683 (6,480-7,735) | +9.6% |
| 32K prompt tok/s | 7,622 (7,620-7,821) | 7,289 (7,287-7,303) | 7,974 (7,828-7,983) | +9.4% |
| agent follow-up turn, first token (s) | 0.72 | 0.76 | 0.64 | -15.5% |
| short chat decode tok/s | 219.4 | 207.6 | 196.5 | -5.3% |
| agent turns decode tok/s | 220.4 | 204.8 | 212.4 | +3.7% |
| 4 clients, total tok/s | 167.6 | 157.9 | 155.8 | -1.3% |

The two decode rows move in opposite directions between the NVFP4 arms, which differ only in the prompt path, so I read them as run-to-run spread (MTP acceptance varies with the answer).

Longer prompts, from an earlier window on 2026-10-14 (the same three arms, each from a cold start, one request per depth through our own client, prompt tok/s as the server reports it):

| prompt tokens | 6.6K | 26K | 52K | 89K | 130K | 179K | 244K |
|---|---:|---:|---:|---:|---:|---:|---:|
| `w4a8` | 3,752 | 6,921 | 7,622 | 7,758 | 7,695 | 7,541 | 7,349 |
| `w4a4` | 4,072 | 7,492 | 8,308 | 8,487 | 8,404 | 8,239 | 8,008 |
| w4a4 vs w4a8 | +8.5% | +8.3% | +9.0% | +9.4% | +9.2% | +9.3% | +9.0% |

**A first-request cost on both NVFP4 arms.** The first fresh 4K prompt and the first fresh 16K prompt read 15-25% slower than the next two of the same size (`w4a8`: 3,804 then 4,957 / 4,965 at 4K, 5,951 then 7,024 / 7,010 at 16K; `w4a4`: 4,218 then 5,698 / 5,603, and 6,480 then 7,683 / 7,735). IQ3_S's first runs were within 2% of its later ones, and the 32K test, which runs after both, had no slow first run on any arm.

**Not covered.** `docs/NVFP4.md` also describes `w4a4x2` and `STRATA_PREFILL_X2_MIN_K`, but this engine accepts only `w4a8`, `w4a4` or `fp16` ([moe_mmq.cu#L188-L196](https://github.com/Niko1221/Strata/blob/296b59d2f73c8f04a805df29c2fa97655adace78/src/prefill/moe_mmq.cu#L188-L196); `main` is at the same commit), so neither ran. Nothing here measures `w4a4`'s answers (the doc's KL figures stand); `fp16` and NVIDIA's own pack were not run; Windows only.