Issues / #2407
#2407 STRATA_PIN_BY_NEED=1 lowers prompt throughput by 11-28% on RTX 4080 SUPER (Windows), opposite to the RTX 5070 result
open · @olituks · 0 Kommentare · Auf GitHub
BenchmarksSetup & installNVIDIA / CUDAModels & quantsWindows
Beschreibung
**Setup:** RTX 4080 SUPER 16 GB (PCIe 4.0 x16), Ryzen 7 7800X3D, 64 GB RAM (every expert in RAM, `arena_mib` 47962), Windows 11, driver 616.92 (CUDA 13.4), Strata 0.1.44.1 (296b59d2), IQ3_S, 196,608-token context, KV q4_0 + `--kv-grow`, calibrated settings, vision encoder on the GPU during these runs. **Results** (`BENCH.bat --long`, 3 runs each, medians, compared with the same config without the opt-ins): | | Baseline | `--adapt-async 1` | `STRATA_PIN_BY_NEED=1` | Both | | --- | ---: | ---: | ---: | ---: | | Chat decode tok/s | 65.7 | 67.1 (+2.1%) | 69.8 (+6.2%) | 74.9 (+14.0%) | | Agent decode tok/s | 68.1 | 66.8 (-1.9%) | 69.7 (+2.3%) | 70.5 (+3.5%) | | 4K prompt tok/s | 1590 | 1591 (0%) | 1416 (-11.0%) | 1412 (-11.2%) | | 16K prompt tok/s | 2125 | 2127 (0%) | 1897 (-10.7%) | 1898 (-10.7%) | | 32K prompt tok/s | 2150 | 2149 (0%) | 1904 (-11.4%) | 1903 (-11.5%) | The prompt loss is very stable (each run within 0.5% of its median), so it does not look like noise. **Observations:** - PR #1905 measured +4.4% prompt and decode within +-1% on an RTX 5070 12 GB. On this card it is the reverse: prompt -11%, decode +2 to +6%. - `--adapt-async 1` alone is neutral here; together with `STRATA_PIN_BY_NEED` the short-chat decode gain is larger (+14%) than with either one alone. - The runs above had the image encoder on the GPU (only 61 MiB of VRAM free). Re-measured with the encoder on the CPU (4552 experts in VRAM, everything else identical), the loss is larger and decode no longer gains: | | Baseline (encoder on CPU) | `STRATA_PIN_BY_NEED=1` | Change | | --- | ---: | ---: | ---: | | Chat decode tok/s | 88.6 | 83.3 | -6.0% | | Agent decode tok/s | 98.1 | 94.2 | -4.0% | | 4K prompt tok/s | 2165 | 1760 | -18.7% | | 16K prompt tok/s | 3456 | 2517 | -27.2% | | 32K prompt tok/s | 3489 | 2522 | -27.7% | | Agent turn TTFT s | 1.85 | 2.11 | +14.2% | So on this machine the setting costs more the more experts the card holds. Is the pinning order (expert profile first) expected to hurt the prompt path when a 16 GB card already holds ~4,000 experts? Happy to test a variant, for example a different `STRATA_ARENA_PIN_GIB`. Benchmark context: #2404, related: #2406.
Mehr auf der Site
Links zu Install, Modellen, Releases.