Issues / #2406
#2406 Image encoder on the GPU costs ~35-44% decode and ~40% prompt speed on a 16 GB card (DETAILS.md says 2-8%)
open · @olituks · 0 comments · View on GitHub
BenchmarksSetup & installNVIDIA / CUDAModels & quantsWindows
Description
**Setup:** RTX 4080 SUPER 16 GB (PCIe 4.0 x16), Ryzen 7 7800X3D, 64 GB RAM, Windows 11, driver 616.92 (CUDA 13.4),
Strata 0.1.44.1 (296b59d2), IQ3_S, 196,608-token context, KV q4_0 + `--kv-grow`, calibrated settings.
**What changed:** only `"vision": {"gpu": true}` -> `"gpu": false` in `strata-iq3_s.json`. Engine args identical.
**Results** (`BENCH.bat --long`, 3 runs each, medians; full report in #2404 ):
| | Vision on GPU | Vision on CPU | Change |
| --- | ---: | ---: | ---: |
| Chat decode tok/s | 65.7 | 88.6 | +34.9% |
| Agent session decode tok/s | 68.1 | 98.1 | +44.1% |
| 4K prompt tok/s | 1590 | 2165 | +36.1% |
| 16K prompt tok/s | 2125 | 3456 | +62.6% |
| 32K prompt tok/s | 2150 | 3489 | +62.3% |
| Agent turn TTFT s | 2.37 | 1.85 | -22% |
Engine facts from the two runs:
| | Vision on GPU | Vision on CPU |
| --- | ---: | ---: |
| `expert_slots` | 4023 | 4552 |
| `expert_cache_mib` | 7862 | 8888 |
| `vram_free_mib` | 61 | 87 |
The vision-on-CPU result was reproduced in a second, separate engine start (prompt throughput within 0.3%, decode
within 5%). A real agent session afterwards (~90K tokens in context) generated at 97 tok/s with an 81.8% expert-cache
hit (72-80% before the change).
**Why I'm reporting it:** DETAILS.md ("Speed with images on") describes the GPU encoder as 2-8% slower, from ~1,000
fewer experts in VRAM. Here the cache gained 529 experts (+13%, about 1 GiB), but speed went up 35-62%. `vram_free_mib`
is almost the same in both runs, so it does not look like the engine simply had more headroom.
Questions:
- Is this expected on a 16 GB card at a long context, for example because `--prefill auto` borrows expert-cache slots
and gets larger chunks with a bigger cache?
- Could the encoder's VRAM (the reservation, or its CUDA context while idle) cost more than its size suggests?
- If it is expected, setup could recommend `--vision cpu` on 16 GB cards and DETAILS.md could mention it.
Happy to run more tests (logs, `--dump-routing`, a specific build) if useful.Related on this site
Quick links to related pages.