#6470 [Engine] CUDA Q8_0 mmproj loses its VRAM saving with the native picture reader

open · @DamonFangCN · 2 Kommentare · Auf GitHub

Setup & installNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

On the same RTX 4090, my Qwen3.8-Flash-Next Q8_0 mmproj used 266 MiB less GPU memory than BF16 with the previous reader. With the native reader in Strata v0.2.17, both formats peaked at 1510 MiB for the vision helper during startup. Both runs used resident GPU vision, a 2048-token cap and the same warm-up shape (2025 image tokens).

| Reader build | BF16 mmproj | Q8_0 mmproj |
| --- | ---: | ---: |
| Engine 0.1.57, llama.cpp/mtmd-based | 1624 MiB | 1358 MiB |
| v0.2.17 / Engine 0.1.63, native CUDA | 1510 MiB | 1510 MiB |

These are per-process NVML startup peaks sampled about every 10 ms in separate source-build runs. The measurements concern the vision helper. The text engine also has local deployment patches. The native helper's CUDA weight loader matches the upstream tag.

The new loader expands Q8_0 matrices to FP16 before uploading them. The 83 Q8_0 matrices in this mmproj take about 314 MiB in GGUF and 592 MiB after FP16 expansion. That accounts for the loss of the resident weight saving. [CUDA weight loader at v0.2.17](https://github.com/Niko1221/Strata/blob/v0.2.17/terminal/tools/vision/own/vit_gpu.cu#L719-L737)

### Smallest steps or command that reproduce it

Use the same NVIDIA GPU for both starts. Set `vision.gpu=true`, `vision.elastic=false` and `vision.max_tokens=2048` in the run config. Start once with the BF16 mmproj, then restart with a Q8_0 mmproj quantized from the same BF16 file. Sample the GPU memory of the `strata-vision` process through warm-up. Run the two cases separately so allocations from one helper cannot overlap the other.

### Reproduction status

Reproduced locally

### Strata version

Strata v0.2.17, Engine 0.1.63, source commit `12d491d`. The v0.2.16 release notes identify the switch to the native picture reader. [v0.2.16 release](https://github.com/Niko1221/Strata/releases/tag/v0.2.16)

### Graphics card, memory and system

NVIDIA RTX 4090 (24 GB VRAM), 64 GiB host RAM (62.37 GiB visible to Linux), Ubuntu 26.04.1 LTS (x86_64).

### Model, size and settings

Qwen3.8-Flash-Next UD-IQ4_XS v3.1 with resident CUDA vision and a 2048 image-token cap.

### Requested behavior

Would an optional CUDA path that keeps Q8_0 weights quantized in VRAM be in scope? On a 24 GB card this space can go back to the expert cache. If FP16 expansion is the intended CUDA tradeoff, please document the GPU memory behavior alongside the CPU recommendation. The earlier CPU recommendation is in #625. #5872 also has CUDA mmproj measurements with the older reader.

Mehr auf der Site

Links zu Install, Modellen, Releases.