#6472 [Benchmark] RTX 4090 24 GB VRAM headroom for long-context KV and IQ4 experts
open · @DamonFangCN · 0 comentarios · En GitHub
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux
Descripción
### What did you measure, or what went wrong On a consumer GPU with 24 GB of VRAM, long-context KV, prefill buffers, and GPU-resident experts share the same budget. I measured three ways to free VRAM on one RTX 4090 running Qwen3.8-Flash-Next with DFlash2 and resident GPU vision. The recovered space can help fit a larger GPU KV allocation, leave room for long-prompt prefill, or keep more experts on the card, depending on the configuration. CUDA module LAZY and the local GPU dense arena are both in use on my Strata 0.2.17 / Engine 0.1.63 build today. GPU Q8_0 vision saved VRAM with the older reader. The new native reader expands those weights and the measured startup saving went to zero (#6470). #### What is running on 0.1.63 The live CUDA engine runs with `CUDA_MODULE_LOADING=LAZY`, `CUDA_MODULE_DATA_LOADING=EAGER` and `STRATA_NATIVE_DENSE_ARENA=1`. Its startup log confirms one arena allocation and byte-for-byte GPU readback of all 301 dense projection matrices. It served a 258,047-token request with a **262,144-token context limit and GPU-resident INT8 KV** (`--kv-resident 0`), alongside **5,666 actual GPU expert slots**. DFlash2, 54K snapshot reuse, and resident BF16 vision at 2048 image tokens also passed. Minimum free VRAM during acceptance was 182 MiB. That build also contains other local changes, including HC Q8. This is a newer-engine capacity and compatibility check. The individual VRAM deltas below come from fixed-slot comparisons on 0.1.57. #### Measured savings on the 4090 | Change | Measured VRAM saving | Comparison | | --- | ---: | --- | | CUDA module LAZY, data EAGER | 318 MiB | Fully warmed, fixed 5,204 actual experts, versus module/data EAGER | | One GPU allocation for dense projections | 254 MiB | Matched warm phases, fixed 5,204 actual experts, arena off versus on | | GPU Q8_0 vision with the older CUDA reader | 266 MiB at startup, 264 MiB during encoding | Same 2048 image-token cap, BF16 versus Q8_0 helper peak | These are separate A/B measurements at a fixed 262,144-token context limit. They measure free VRAM. A different GPU KV allocation or context limit would need its own test. #### CUDA module and data LAZY All three arms used the same locally patched Engine 0.1.57 binary, the dense arena, 5,204 actual GPU experts, and fixed expert-cache membership. NVML sampled the whole GPU at about 10 ms intervals. The fully warmed samples were taken after vision, 54K snapshot save/restore and the near-256K test. All graph widths from 1 through 8 were captured in every arm. | `CUDA_MODULE_LOADING` / `CUDA_MODULE_DATA_LOADING` | Free at Ready | Free fully warmed | Time to Ready | First text TTFT | | --- | ---: | ---: | ---: | ---: | | EAGER / EAGER | 730 MiB | 530 MiB | 60.58 s | 0.67 s | | LAZY / EAGER | 1026 MiB | 848 MiB | 60.57 s | 0.61 s | | LAZY / LAZY | 1026 MiB | 848 MiB | 74.59 s | 3.97 s | Module LAZY's saving persisted after warm-up. Data LAZY added 0 MiB on this workload and its one cold run took longer. For EAGER/EAGER versus LAZY/EAGER, median Prefill/Decode was 2753/82.7 versus 2744/82.6 tok/s at 8K and 2951/83.3 versus 2930/82.7 tok/s at 32K. The short general-text Decode median fell from 84.4 to 75.2 tok/s. Three samples per fixture and one cold start per arm in fixed order limit the timing conclusions. Strata sets EAGER when the caller has not set `CUDA_MODULE_LOADING` to reduce the risk of loading kernel code into a nearly full GPU mid-prompt. Our measured use is an explicit opt-in with reserve and warm-up checks. [Upstream startup code](https://github.com/Niko1221/Strata/blob/v0.2.17/engine/src/program/generate.cpp#L3701-L3711) #### GPU dense projection arena The local opt-in `STRATA_NATIVE_DENSE_ARENA=1` places the existing quantized dense projections in one GPU allocation at 512-byte-aligned offsets, with separate scratch. It keeps their GGUF bytes and tensor types. GPU readback matched all 301 matrices (2,581,928,960 payload bytes) byte for byte on this UD-IQ4_XS v3.1 layout. With the same binary, `--expert-cache 4452` produced 5,204 actual slots both ways. Module LAZY and data EAGER stayed fixed. The arena raised the minimum free VRAM from 628 to 882 MiB and gave the same 254 MiB gain at each matched idle measurement. Median Prefill/Decode was 2745/104.5 versus 2752/104.3 tok/s at 8K and 2937/157.1 versus 2930/157.1 tok/s at 32K. Six of twelve paired final texts matched exactly, including all six long-prompt outputs. The short text and code outputs differed. The byte check covers uploaded weights, and the output comparisons cover these prompts. A separate arm enabled the arena and `--expert-cache auto` with the existing 700 MiB reserve. It reached **5,311 actual slots**, 107 above the fixed comparison, and passed DFlash2, 54K snapshots, the near-256K request, GPU vision and zero routed expert reads from disk. #### Resident GPU Q8_0 vision | CUDA vision helper | BF16 mmproj | Q8_0 mmproj | | --- | ---: | ---: | | 0.1.57 mtmd helper, startup peak | 1624 MiB | 1358 MiB | | 0.1.57 mtmd helper, encoding peak | 1630 MiB | 1366 MiB | | 0.1.63 native helper, startup peak | 1510 MiB | 1510 MiB | These are per-process NVML peaks on the same card with a 2048 image-token cap. The new CUDA helper expands Q8_0 to FP16 at upload, so saving disk space with Q8_0 no longer reduced the helper's measured startup VRAM here. The native-helper rows are startup measurements from separate runs. They leave matched encode latency and OCR quality open. #6470 has the reproduction and the request for an optional quantized CUDA path. #### Setup and scope RTX 4090 24 GB (sm_89, 350 W limit), Ryzen 9 7950X, 64 GiB RAM, Linux x86_64. The fixed-slot comparisons used a patched Strata 0.2.11 / Engine 0.1.57 source build based on `c3b3f50`. The native vision comparison used Strata 0.2.17 / Engine 0.1.63 source commit `12d491d`. The model was [Unsloth Qwen3.8-Flash-Next UD-IQ4_XS v3.1](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-IQ4_XS). The two fixed-slot tests launched the local server with the same IQ4 pack, four GGUF shards and expert heat profile within each comparison. Relevant native arguments were `--expert-cache 4452 --max-context 262144 --kv int8 --kv-resident 0 --prefill auto --resident-experts --vision --vram-reserve-mib 700 --pcie-frac 0.20 --draft dflash2 --spec 6`. Keep resident vision at 2048 image tokens. Set `--adapt-every 0` for the three CUDA loading modes. The arena A/B used `--adapt-every 1` in both arms and changed only the arena and its startup readback switch. The text tests used synthetic prompts, temperature 0, seed 1234 and reasoning off. The actual inputs were 72, 78, 8200 and 32775 tokens. The new-token caps were 384 for short text and code and 256 for 8K/32K. Would a narrowly scoped, opt-in CUDA dense arena be useful upstream for single 24 GB cards? It would give users room to fit their target context and maintain more GPU expert residency within the same VRAM budget. The fixed-slot results also give a data point for documenting module LAZY as an opt-in. The measurements cover one RTX 4090 and the paths exercised above. ### Reproduction status Measured locally
En el sitio
Enlaces a install, modelos, releases.